SOPERA Blog

SOPERA Support for EDI and EDIFACT

Recently, we’ve discovered - thanks to a number of customer requests - that support for EDI and EDIFACT is a must-have feature. Currently, with our SOPERA DI we are delivering the first version of our EDI/EDIFACT related components, based on an open source smooks framework. These components work quite well for simple EDI messages. You can configure the EDI mapping and read EDI messages into your SOPERA DI Jobs. We have already used them in a few customer cases and have learned quite a lot from it. Here are some of the major issues we encountered:

• You need to specify the mappings manually, even when processing standard EDI messages, for example, UN EDIFACT.
• It’s hard to read the UN/EDIFACT interchange that contains multiple messages of the same type, and it’s impossible to read UN/EDIFACT interchange with multiple messages of different type.
• The current state of components is not capable of writing out EDI files, only parsing/reading is supported to date.
• The tooling provided to read EDI messages inside the SOPERA DI needs to be improved. For example, we could display the message structure to simplify the mapping with a point-and-click feature.

At the same time the framework we were previously using for parsing EDI messages in SOPERA DI EDI components has made significant progress in the areas mentioned above allowing us now to take advantage of these improvements as well as to continue with our efforts to test and fix bugs. Here I would like to give you a short preview of the improvements we’ve made in this area.

UN/EDIFACT Mappings

The first area we improved is the manual definition of EDIFACT mapping files. This task can be quite tedious - one has to convert the UN/EDIFACT definitions into an XML-based mapping that can be used for EDI parsing. With the updated EDIFACT tooling you can get them automatically generated from UN definitions. Let us take for example the UN Customs cargo report message (CUSCAR). The picture below demonstrates this process:

Automated Mapping Generation

Here we take the CUSCAR definition which is available for download for example from within the package d99a.zip, and automatically generate XML Mappings according to the CUSCAR description from the UN. Now you can be sure all segments, segment groups, fields, and components are available in the mapping with all cardinalities as well as optional and mandatory flags. Moreover, since the ZIP file from the UN contains not only a single CUSCAR dialect definition but many others you will also get them packaged and ready to use.

UN/EDIFACT Inter-exchange processing

As you might know the EDI inter-exchange may contain multiple EDI messages of different types, each messages starts with the header segment that defines the message type (dialect), release year etc. For example, here is the header for the CUSCAR message:

UNB+UNOA:2+SENDER+SENDER+100421:0437+1918'
UNH+163477+CUSCAR:D:99A:UN:SCPRBL'

The second field of the UNH segment defines the dialect - here it’s CUSCAR released in the first part of the year 1999 (D99A) by the UN. The updated UN/EDIFACT parser is able to recognize individual messages as well as message dialects and dynamically chooses the appropriate mapping from the mapping JAR files generated before, as shown here:

Dynamic UN/EDIFACT Interexchange parsing

Writing out UN/EDIFACT files

Currently, our SOPERA DI EDI components do not support EDIFACT file generation. Basically, EDIFACT files are just simple text files, and one could simply use a plain template approach to generate EDI. However, we’d like to have additional semantic and syntax checks as well as validation checks. For this purpose we could use an EJC compiler as shown in this figure:

EDI-to-Java roundtrip

The basic idea is that we could generate a set of Java classes based on the UN/EDIFACT mapping files. These classes would then represent segments, segment groups or fields/components specified in the mapping file, and the best part is- you can use them to read EDI as well as write EDI out. It’s similar to the Java API for XML Data Binding, but in this case we bind the Java objects to the EDI content.

Improved UI for SOPERA DI Components

All of the features mentioned above are very useful but our experience shows us that we need to improve the user interface of SOPERA DI EDI components. Right now we are still in the planning and investigation phase. So if you think you have a nice feature proposal we would be glad to hear from you, just leave your comment on this blog post.

Conclusion

All in all with the new functions of the extended EDI processing we can provide a significantly improved user experience that will greatly simplify EDI processing. However, there is still some work to do, but what is clear right now is that all of it will be brought to you by SOPERA and the smooks team as an open source project so you are welcome to participate, contribute, and improve it.

Renat Zubairov Development, Strategy EDI, EDIFACT, smooks, SOPERA, SOPERA DI, Talend

We have new partners

Last year we launched the new SOPERA Partner Program. Our partners get earlier access to new product versions and technologies. They have the opportunity to give input to the SOPERA roadmap and release plans, and benefit from best practices in dedicated areas of application. Trainings to increase the technical know-how are included in the partner fee.

New Partners

Today we welcome our new partners:

• CapGemini
• CSC
• cologne intelligence
• saxonia
• senacor
• tarent
• USU AG

You can find more information about our Partner Program here.

Renat Zubairov Events, Management, Strategy Partner Program, Schlagwort hinzufügen, SOPERA Partners

Things have been rather quiet around the Eclipse SOA Intitative. The cause for that lies squarely with the exit of some commercial software vendors from the Industry Working Group shortly before it was to start in the fall of 2009.

I think that speaks volumes about the insecurity that these vendors are facing when dealing with Open Source projects. For the others, like Obeo, itemis & SOPERA whose business models are closely entwined with Open Source projects, the work has gone on. Recently, Engineering, Italy’s biggest system integrator joined the Industry Group in February 2010 and has offered up two new SOA projects: eBPM und eBAM giving the whole initiative a new momentum. As a result of which, half a year later, the Eclipse SOA Industry Working Group will now attend the SOA days at the Deutsche Post officially with a keynote presentation by Mike Milinkovich.

Ricco Deutscher Development, Management, Strategy eBAM, eBPM, Eclipse SOA

Extending our Reach

SOPERA’s SOA open source software is based on a building a community for developers. SOPERA has created a partner program to extend our ability to reach members of this community. Our partner pogram offers strategic alliances and enables us to offer customized and innovative solutions.

Types of Partnerships

SOPERA offers a partner program with various options for cooperation. Here is a short list of what our partner program covers:

• System Integrators
• Master Resellers
• Independent Software Vendors
• Technology Partners

For more information, see Partner Program.

Anne Aloysious Events, Strategy partner

The Eclipse SOA Initiative has entered now the official 30 day proposal phase. The charter can be seen here. So far the following members have given its official commitment (in alphabetical order):

• itemis (Steering Committee, represented by Wolfgang Neuhaus)
• obeo (Steering Committee, represented by Stephane Drapeau)
• Progress Software (represented by Oisin Hurley)
• Software AG (represented by Prasad Yendluri)
• SOPERA (Steering Committee, represented by Ricco Deutscher)

The Initiative will start work after the official proposal phase. Until then, a few things still need to be done, like

• aligning the requirements to fit the first two official milestones
• providing the first complete Eclipse SOA platform package

It’s clear that the first Eclipse SOA Platform package will be small in size, to begin with.

The strength of the Eclipse Foundation – the IP security – can at times also be an obstacle. For that reason, the BPEL-Engine Apache ODE cannot be delivered together with the Eclipse SOA Platform package. Therefore SOPERA will provide a member distro, which will enhance the functionality of the Eclipse SOA Platform by including a process orchestration component. On the other hand, the members also want to ensure that the functionality on the tooling and runtime sides increase at the same rate and with mutual support. Already today there is an Eclipse SCA editor available. Swordfish does not support SCA on the runtime yet and therefore SCA will be part of the platform at a later stage. So Obeo will provide a member distro based on Tuscany.

Ricco Deutscher Strategy

I’m back from the first Eclipse Board Meeting in Europe, which took place in Berlin. There, I had the opportunity to present the latest news from the Eclipse SOA Initiative to the Board (find a copy here). Since our last announcement at the EclipseCON in March this year, the initiative has incredibly evolved. What started as an initiative by SOPERA, has turned into an initiative by the Eclipse ecosystem, as two things unfolded unexpectedly:

1. To my surprise, other Eclipse members also share our vision and have signalized their participation, and
2. The foundation provides a new type of organizational construct - the Industry Working Group (IWG), which allows to map the initiative’s goals to an appropriate organizational structure. This new organizational construct, the Industry Working Group, allows to setup sub-consortiums within the Eclipse consortium with own governance rules and brandings.

Once I realized that, I’m really fascinated and convinced, that it brings a lot of value to the Foundation itself (Mike – that credits belong to you. Great job!)

Back to the Eclipse SOA Initiative. The goals remain unchanged:

• Deliver an integrated and extensible SOA framework based on Equinox, that includes a tooling platform and a runtime framework
• Foster adoption of this framework by vendors and system integrators
• Achieve interoperability between products provided by the participating vendors

These goals can be achieved through two measures in the Foundation:

1. Create a new top-level project “Eclipse SOA Platform“, which initially will consist of the two existing projects: (a) the STP projects for the tooling, and (b) the Swordfish project for the runtime. This new project should be the home for all future SOA relevant projects inside the Foundation(e.g. BPM, BAM, Registry/Repository). In the meanwhile, a new charter was created specially for that purpose. As next step, the charter will be approved by the board.
2. Found the Eclipse SOA Instustry Wokring Group (We are looking for a cool nick name, so: Keep me posted if you have an idea!), which will be responsible for 3 things. (a) It will specify the requirements of the Eclipse SOA Platform, (b) it will create and take care of a new brand for that platform, and (c) it will define measurable criteria for the usage of that brand, as the brand should also be used by non-members, namely ISVs and system integrators.

This was a lot of work for the past 3 months, and it was a great teamwork together with Oisin Hurley from Progress, and Donald Smith from the Eclipse Management Organization (Guys, it was a great pleasure to work with you!). I also want to thank Ralph Müller for his support (Ralph -  I very well remember the meeting we had with Mike at the Frankfurt Airport, thank you for your patience with me!). The next weeks will unveil, who will by the founders of this IWG beside SOPERA and Progress. But since this last board meeting, it is very clear: we already reached the critical mass. Who they will be, will be published here very soon, so: Stay tuned!

Ricco

Ricco Deutscher Strategy

SOPERA will announce its Eclipse SOA initiative (see homepage and press release) tomorrow at the EclipseCON in Santa Clara, CA. With this initiative, SOPERA will develop an integrated, coherent SOA platform inside Eclipse. As of today, there are only individual SOA components available through different, widespread, uncoordinated Eclipse projects like Swordfish, STP, and BPEL. With the Eclipse SOA initiative, there will be an integrated Eclipse package that can be downloaded from the Eclipse Website to create service oriented applications. Part of this initiatve will be the development of a new Service Registry/Repository in the context of a new Eclipse project. Currently, all available open source registry/repository projects are not qualified for enterprise level usage.

Even though the Eclipse SOA initiative is actually a SOPERA initiative, SOPERA is looking for cooperation from other SOA-interested companies and Eclipse members. Such cooperation will guarantee that the emerging Eclipse SOA platform will be developed closer to the market needs and can be available even sooner. There was a discussion with the EMO (Eclipse Management Organization) yesterday evening to decide on an organizational structure into which the Eclipse SOA initiative can be embedded. It seems that an “Industry Working Group” will provide the perfect organizational structure for this initiative, as flexible governance structures can be defined individually as needed.

We do not expect that traditional license-oriented SOA vendors effusively praise the Eclipse SOA initiative, as this could end up in a market cannibalization effect for the traditional license-oriented SOA vendors and their commercial platforms. At the EclipseCON we will have our first discussion with Eclipse members, and it is not surprising that we have already received greatest interest from SOA user companies. SOPERA is in contact with some major customers that are more than willing to back such an initiative.

Ricco Deutscher Strategy

Under the umbrella of the OSBF (Open Source Business Foundation), SOPERA, Microsoft, 1&1, Corisecio, and OpenXchange presented at the Cebit 2009, a path-breaking prototype of a heterogeneous application cloud based on an Internet Service Bus (see the OSBC press release). This announcement is a major milestone, and represents a new way in which business applications can be deployed through enterprises in an absolutely flexible and cost-efficient way.

Today, the application landscapes of enterprises are operated through dedicated data centers, but in the future, parts of it will be provided as a service from an internet cloud, from multiple providers. It will start with commodity services, such as, archiving and e-mailing, but will soon flash over core applications, such as, CRM. Enterprises that will not follow this trend will directly experience a competitive disadvantage, as their IT will quickly become too inflexible and expensive.

But also in such a scenario we face the fundamental integration question. Cloud computing has additional exigencies beyond the traditional application integration requirements. The fact that services can be obtained from different providers, makes service level monitoring indispensable. Furthermore, have the different platforms on which the services are being developed and operated (basically Java and .Net), to be absolutely transparent. Enterprises will use more than one provider for one, single service, and the provider selection will occur dynamically through the service levels required at that point of time. The Internet Service Bus - the Enterprise Service Bus of the application cloud – will have to meet all this requirements. SOPERA has proved at the Cebit 2009 that its prototyped fulfills already today core requirements of an Internet Service Bus, including

• A decentralized bus architecture that ensures scalability in the cloud,

• A native implementations in the predominant application platforms Java and .Net, and

• A dynamic, policy-driven, mediation between service-provider and service-consumer.

With it, SOPERA is at an excellent position for the future cloud computing market!

Ricco Deutscher Strategy