Recently, while working on a project, we were working on importing content from CSV files. We were using Drupal’s Migration API to achieve this, as it provides a fully featured framework to import/migrate content.
During the content migration process, when a particular content item is being migrated, we wanted to find out whether it has been previously migrated based on one of the fields.
To do this, during the migration process, we would need to look for that field value in the destination site’s node, and return that node id if that value is present.
We implemented this in the following steps:
- Implementing our own ID map plugin
- Telling migration templates or migration config to use our newly created plugin.
What are ID Map plugins?
Migration’s ID Map plugins keep track of relationships between source content and migration content.
Typically, when we write a migration, we provide mapping with a source ID. Source ID is a unique ID in the source database, CSV, or any other source format.
If you look at the migration template here, you will notice the “Keys” attribute in the “source” mapping block.
Here, “keys” specifies a unique ID in the source CSV file. So, ID Map plugin will maintain a relationship between “Item-ID” and the migrated node’s nid. Typically this relationship is stored in tables starting with migrate_map_.
For example, if you have a migration with the ID node_product, the table name will be something like migrate_map_node_product.
Implementing ID Map plugin
If you notice the default “id_map” plugin here, Drupal\migrate\Plugin\migrate\id_map, you will come across a few functions that are helpful to us: lookupSourceID(), lookupDestinationId(), lookupDestinationIds(). These functions typically help us to find source ID and destination ID based on a given value.
As we want to check whether the current node is migrated or not, we need to use lookupDestinationId().
To implement our own plugin, we should create a new plugin in our module’s namespace. The class for our new plugin should extend “Sql” class (Sql class is located here: Drupal\migrate\Plugin\migrate\id_map). That way, we can use all the other remaining functionalities of the default Sql id_map plugin.
Let’s say your module is my_module. We'll name our plugin class “ContentUpdate”.
Create a new file called “ContentUpdate.php” in the “Drupal\my_module\Plugin\migrate\id_map" namespace.
Override the “lookupDestinationId()” method given below.
After that, override the lookupDestinationIds() method in our “ContentUpdate.php” class so we can write our business logic to find out whether the content is migrated or not.
You can implement your business logic to find the node ID based on a given source value. Partial implementation is given below.
In this way, we can look for existing IDs from our destination site. Refer to this gist for the entire snippet.
Use the new plugin in migration templates
Now that we have created a new plugin, it’s time to use this in our migration templates.
In our migration templates, we can specify the id_map plugin using the “idMap” key.
See the following snippet to see how we can specify.
Refer to this gist for the full migration template.
While we saw one use case of the ID Map plugin, there are several other usages for ID Map plugins. When you are migrating data from a non-SQL database and you have a source database key that is pretty big (bigger then 3072 bytes), you might need to alter the schema and change the width of the table field. You can read about that in the blog post by Matt Glaman here.