EDI and EDIFACT support in SOPERA DI
SOPERA Support for EDI and EDIFACT
Recently, we’ve discovered - thanks to a number of customer requests - that support for EDI and EDIFACT is a must-have feature. Currently, with our SOPERA DI we are delivering the first version of our EDI/EDIFACT related components, based on an open source smooks framework. These components work quite well for simple EDI messages. You can configure the EDI mapping and read EDI messages into your SOPERA DI Jobs. We have already used them in a few customer cases and have learned quite a lot from it. Here are some of the major issues we encountered:
- You need to specify the mappings manually, even when processing standard EDI messages, for example, UN EDIFACT.
- It’s hard to read the UN/EDIFACT interchange that contains multiple messages of the same type, and it’s impossible to read UN/EDIFACT interchange with multiple messages of different type.
- The current state of components is not capable of writing out EDI files, only parsing/reading is supported to date.
- The tooling provided to read EDI messages inside the SOPERA DI needs to be improved. For example, we could display the message structure to simplify the mapping with a point-and-click feature.
At the same time the framework we were previously using for parsing EDI messages in SOPERA DI EDI components has made significant progress in the areas mentioned above allowing us now to take advantage of these improvements as well as to continue with our efforts to test and fix bugs. Here I would like to give you a short preview of the improvements we’ve made in this area.
UN/EDIFACT Mappings
The first area we improved is the manual definition of EDIFACT mapping files. This task can be quite tedious - one has to convert the UN/EDIFACT definitions into an XML-based mapping that can be used for EDI parsing. With the updated EDIFACT tooling you can get them automatically generated from UN definitions. Let us take for example the UN Customs cargo report message (CUSCAR). The picture below demonstrates this process:
Automated Mapping Generation
Here we take the CUSCAR definition which is available for download for example from within the package d99a.zip, and automatically generate XML Mappings according to the CUSCAR description from the UN. Now you can be sure all segments, segment groups, fields, and components are available in the mapping with all cardinalities as well as optional and mandatory flags. Moreover, since the ZIP file from the UN contains not only a single CUSCAR dialect definition but many others you will also get them packaged and ready to use.
UN/EDIFACT Inter-exchange processing
As you might know the EDI inter-exchange may contain multiple EDI messages of different types, each messages starts with the header segment that defines the message type (dialect), release year etc. For example, here is the header for the CUSCAR message:
UNB+UNOA:2+SENDER+SENDER+100421:0437+1918' UNH+163477+CUSCAR:D:99A:UN:SCPRBL'
The second field of the UNH segment defines the dialect - here it’s CUSCAR released in the first part of the year 1999 (D99A) by the UN. The updated UN/EDIFACT parser is able to recognize individual messages as well as message dialects and dynamically chooses the appropriate mapping from the mapping JAR files generated before, as shown here:
Dynamic UN/EDIFACT Interexchange parsing
Writing out UN/EDIFACT files
Currently, our SOPERA DI EDI components do not support EDIFACT file generation. Basically, EDIFACT files are just simple text files, and one could simply use a plain template approach to generate EDI. However, we’d like to have additional semantic and syntax checks as well as validation checks. For this purpose we could use an EJC compiler as shown in this figure:
EDI-to-Java roundtrip
The basic idea is that we could generate a set of Java classes based on the UN/EDIFACT mapping files. These classes would then represent segments, segment groups or fields/components specified in the mapping file, and the best part is- you can use them to read EDI as well as write EDI out. It’s similar to the Java API for XML Data Binding, but in this case we bind the Java objects to the EDI content.
Improved UI for SOPERA DI Components
All of the features mentioned above are very useful but our experience shows us that we need to improve the user interface of SOPERA DI EDI components. Right now we are still in the planning and investigation phase. So if you think you have a nice feature proposal we would be glad to hear from you, just leave your comment on this blog post.
Conclusion
All in all with the new functions of the extended EDI processing we can provide a significantly improved user experience that will greatly simplify EDI processing. However, there is still some work to do, but what is clear right now is that all of it will be brought to you by SOPERA and the smooks team as an open source project so you are welcome to participate, contribute, and improve it.