SOPERA Blog

SOPERA Support for EDI and EDIFACT

Recently, we’ve discovered - thanks to a number of customer requests - that support for EDI and EDIFACT is a must-have feature. Currently, with our SOPERA DI we are delivering the first version of our EDI/EDIFACT related components, based on an open source smooks framework. These components work quite well for simple EDI messages. You can configure the EDI mapping and read EDI messages into your SOPERA DI Jobs. We have already used them in a few customer cases and have learned quite a lot from it. Here are some of the major issues we encountered:

• You need to specify the mappings manually, even when processing standard EDI messages, for example, UN EDIFACT.
• It’s hard to read the UN/EDIFACT interchange that contains multiple messages of the same type, and it’s impossible to read UN/EDIFACT interchange with multiple messages of different type.
• The current state of components is not capable of writing out EDI files, only parsing/reading is supported to date.
• The tooling provided to read EDI messages inside the SOPERA DI needs to be improved. For example, we could display the message structure to simplify the mapping with a point-and-click feature.

At the same time the framework we were previously using for parsing EDI messages in SOPERA DI EDI components has made significant progress in the areas mentioned above allowing us now to take advantage of these improvements as well as to continue with our efforts to test and fix bugs. Here I would like to give you a short preview of the improvements we’ve made in this area.

UN/EDIFACT Mappings

The first area we improved is the manual definition of EDIFACT mapping files. This task can be quite tedious - one has to convert the UN/EDIFACT definitions into an XML-based mapping that can be used for EDI parsing. With the updated EDIFACT tooling you can get them automatically generated from UN definitions. Let us take for example the UN Customs cargo report message (CUSCAR). The picture below demonstrates this process:

Automated Mapping Generation

Here we take the CUSCAR definition which is available for download for example from within the package d99a.zip, and automatically generate XML Mappings according to the CUSCAR description from the UN. Now you can be sure all segments, segment groups, fields, and components are available in the mapping with all cardinalities as well as optional and mandatory flags. Moreover, since the ZIP file from the UN contains not only a single CUSCAR dialect definition but many others you will also get them packaged and ready to use.

UN/EDIFACT Inter-exchange processing

As you might know the EDI inter-exchange may contain multiple EDI messages of different types, each messages starts with the header segment that defines the message type (dialect), release year etc. For example, here is the header for the CUSCAR message:

UNB+UNOA:2+SENDER+SENDER+100421:0437+1918'
UNH+163477+CUSCAR:D:99A:UN:SCPRBL'

The second field of the UNH segment defines the dialect - here it’s CUSCAR released in the first part of the year 1999 (D99A) by the UN. The updated UN/EDIFACT parser is able to recognize individual messages as well as message dialects and dynamically chooses the appropriate mapping from the mapping JAR files generated before, as shown here:

Dynamic UN/EDIFACT Interexchange parsing

Writing out UN/EDIFACT files

Currently, our SOPERA DI EDI components do not support EDIFACT file generation. Basically, EDIFACT files are just simple text files, and one could simply use a plain template approach to generate EDI. However, we’d like to have additional semantic and syntax checks as well as validation checks. For this purpose we could use an EJC compiler as shown in this figure:

EDI-to-Java roundtrip

The basic idea is that we could generate a set of Java classes based on the UN/EDIFACT mapping files. These classes would then represent segments, segment groups or fields/components specified in the mapping file, and the best part is- you can use them to read EDI as well as write EDI out. It’s similar to the Java API for XML Data Binding, but in this case we bind the Java objects to the EDI content.

Improved UI for SOPERA DI Components

All of the features mentioned above are very useful but our experience shows us that we need to improve the user interface of SOPERA DI EDI components. Right now we are still in the planning and investigation phase. So if you think you have a nice feature proposal we would be glad to hear from you, just leave your comment on this blog post.

Conclusion

All in all with the new functions of the extended EDI processing we can provide a significantly improved user experience that will greatly simplify EDI processing. However, there is still some work to do, but what is clear right now is that all of it will be brought to you by SOPERA and the smooks team as an open source project so you are welcome to participate, contribute, and improve it.

Renat Zubairov Development, Strategy EDI, EDIFACT, smooks, SOPERA, SOPERA DI, Talend

We have new partners

Last year we launched the new SOPERA Partner Program. Our partners get earlier access to new product versions and technologies. They have the opportunity to give input to the SOPERA roadmap and release plans, and benefit from best practices in dedicated areas of application. Trainings to increase the technical know-how are included in the partner fee.

New Partners

Today we welcome our new partners:

• CapGemini
• CSC
• cologne intelligence
• saxonia
• senacor
• tarent
• USU AG

You can find more information about our Partner Program here.

Renat Zubairov Events, Management, Strategy Partner Program, Schlagwort hinzufügen, SOPERA Partners

Helios and Swordfish

It’s been a while since my last blog post here. We’ve been quite busy these last few days with the Eclipse 3.6 Helios release. We’ve contributed a very nice feature to the Swordfish SOA Runtime Framework Project: remote deployment. Using this new feature you can deploy locally developed Swordfish services into a remote OSGi server.

Swordfish Remote Deployment Dialog

This is a pretty fancy feature because with it you can take your OSGi bundles, pack them into a feature, and deploy them with one click onto a remote system. Behind-the-scenes, there is a complicated set of plug-in packaging and Eclipse p2 magic which makes sure that all the bundles are deployed in a consistent and atomic manner. Read the Swordfish New and Noteworthy for more information.

SOPERA SOA Tutorial

In this releases we’re also offering you a SOPERA SOA ASF tutorial. We have been working on Swordfish for over a year now, and many people have been asking us how Swordfish integrates itself within the SOPERA ASF product stack. Now we can show you how it works.
Just download the Eclipse SOA package and within the Eclipse IDE GUI, select Help > SOA > Install New Tutorials as shown here:

Eclipse SOA Tutorial

Select SOPERA SOA Tutorial: Rent a Car (OSGi) from the list.

Next, simply follow the instructions shown in the GUI.

The tutorial will demonstrate how to use the pluggable nature of Swordfish Runtime to integrate a standalone ESB Runtime into an enterprise-wide SOA infrastructure based on the SOPERA ASF.

Try it out and send us your feedback. If you have questions or need help, post your enquiries in our forum.

Renat Zubairov Development

Many people working in SOPERA (including our CTO) are using Apple Macs. Around 30% of the visitors to our website are also Mac users. However, so far we haven’t had a native Mac OS X installer for SOPERA ASF products. This doesn’t mean that SOPERA ASF will not work on a Mac, however we don’t have it yet as part of our certified platform.

After a couple of requests from our customers I created two simple “How Tos” that show you how to install SOPERA ASF on Mac OS X:

• Mac OS X SOPERA Infrastructure QuickStart - How to install SOPERA ASF infrastructure/SBB
• Mac OS X SOPERA Tooling Quickstart - How to install SOPERA ASF Tooling (so far only Admin Tool)

If you want to see more on this subject, send us your comments here.

Renat Zubairov Development

Embracing Change

I am absolutely excited about open source software projects and the culture it nurtures. Not only is it exciting from the aspect of knowledge sharing and the free exchange of ideas but, mostly what fascinates me is the way in which people are starting to embrace the change and are adopting on open source mindset within their work environment.

Daytime Jobs and Full-time Committers

It has been an interesting period, watching open source project committers, (think about Eclipse, Linux, and Apache) people with daytime jobs and bills to pay volunteering and using their free, leisure hours contributing to a pet project on a regular basis. Just check the mail list traffic or, replies in forums that were created very, very late at night, or even way past midnight. What is especially striking is that sometimes my questions in the open-source project mail lists were responded to much faster and in a much more thorough fashion than for example, when using a ticketing system of some large software vendors where we actually had a paid account!

SOPERA: We, the Open Source SOA Company

As an open source company we are also adopting open source practices. In the last year we opened a forum, we have a download page that does not compel you to put up with a tedious registration process. Our Eclipse activities, as well a number of open source projects on Sourceforge and Google Code are purely open source projects. All these activities are dedicated towards enabling open communication with our users, to empower them with more than adequate, know-how and it is also aimed at getting the necessary feedback from our community.

Culture Shock

As you might well have guessed we are not quite where we want to be, yet. Right now we are working on a public bug tracking project, a public source code repository, and in the future may be even a public wiki. By doing this, we are also changing our internal corporate culture. It is very interesting to observe these changes because honestly, it is really hard to switch from “closed” to “open”.

Don’t Miss the Bus

Think about Wikipedia versus, the Encylopedia Brittanica. Both are online, both are free (the latter, is almost free), but one is booming and the other is fading away. Wikipedia has an open editing model where literally everyone on the web is able to read, modify, and create new articles. Skeptics will definitely say “oh, these evil internet users, just give them the rights and they will destroy everything….” or something like “dumb teenagers can’t write…”, and you may have hit on something true but only partially.

The Wikipedia (and many other wiki-like websites) example shows that the “open” model works, and can be very successful. Note that the open editing model is backed-up by very simple, technical features. For example, in the wiki editing model all modifications are reversible, nothing gets lost nor can it be irrevocably deleted. There will be always a previous version available. As a groupware system the wiki is very well organized to increase awareness about changes, for example, E-mail notifications is a standard feature in almost all wikis. So what we’re saying is; hop on, don’t miss the bus.

Shedding the Closed Source Project Mindset

Moving from the “all-closed” to an open source model is really hard work. Discussions about: “Who can access this data?” or, “What roles shall we have in our bug tracking?”, “Shall we forbid commits during the end game?” will remain and they definitely should be discussed. However, what is really important is where are you coming from? What is your take on this? What is your initial assumption? It can either be “open” up or “close” the doors. Do you see your community, contributors, users,and employees by default as smart, cooperative and committed or dumb, destructive and lazy?

There will be always be mistakes, there is no one-size-fits-all solution. And if you are really, really, successful in your open communication and your community grows, you might have to endure a novel luxury problem - malicious members. However as with the wiki model there are simple technical means to deal with it, think about source control, backups, monitoring, reviewing, and the list goes on.

Are you ready?

I recently read a good article written by Chirs Grams - three signs your corporate culture isn’t ready for the open source way. This article does not tell you how to fix it and this, in my opinion, can only be done iteratively with continuous improvements based on the feedback from our customers and our community.

Below you can submit your vote on what you think is missing from our website. You don’t need to register to vote, so if you are reading this now, please take a minute and vote. Your feedback is extremely important to us.

Renat Zubairov Development open source culture

As promised in the last blog post on SOPERA Intalio|BPM Adapter here comes the first in the series of our blog posts dedicated to the topic of BPM and SOPERA.

Browsing the SOPERA Intalio|BPM Adapter Tooling User Interface

This short video takes you through the user interface and shows you the major functions that we at SOPERA contributed to the standard Intalio|Designer User Interface, such as:

• SOPERA Service Registry View
• SOPERA Project Preferences Page
• SOPERA Intalio|BPM Adapter Eclipse Help

Creating a simple BPM Process

Now, that you have familiarized yourself with the user interface it’s time to try creating a simple BPM process. This video will show you just how easy it is to do that!

As you can see all you need to do to use services from SOPERA Service Registry is to drag an operation from the Service Registry View and drop it onto the BPM diagram.

Adding Security Features

In Intalio|Designer you can implement security features: to do this you simply locate the SOPERA ESB tab in the Properties View of the Designer and then add a policy and authentication credentials as required. This video show you how we do that in less than a minute.

Using Dynamic Credentials

You can also make use of dynamic credentials in SOPERA Intalio|BPM Adapter; here’s how:

We’ll be updating this post in the near future so do come back & visit this page for the latest news on SOA & BPM.

Renat Zubairov Development BPM, Intalio, SOA, SOPERA

Milestone!

Last week we had a very important milestone in the development of SOPERA Intalio|BPM Adapter: we published a Release Candidate for Version 1.0. Currently, it’s not available for the general public to download. Instead, we are going to install for selected customers who will work with it under constant supervision and monitoring from our Professional Services team members. However, in a few of weeks you will all be able to download the first release from our website.

Release 1.0

As you might have seen in our previous blogs there has been a delay in the planned release date of our first release candidate. One of the interesting aspects of our development scenario was that it was accomplished in close cooperation with our selected customers, which allowed us to build many more features that we had at first anticipated during the initial planning phase and this naturally ended up costing us more time. The time was well spent and I’m very satisfied with the result. SOPERA Intalio|BPM Adapter, Release 1.0 contains many useful features and provides a significantly better user experience than we had imagined at the beginning. We enhanced both the runtime and design aspects of Intalio|BPM. Now, BPM users can make the best use of all the benefits found in an enterprise-scale SOA platform provided by SOPERA.

SOPERA Intalio|BPM Adapter

SOPERA ASF and Intalio|BPM

SOPERA ASF and Intalio|BPM in SOPERA Intalio|BPM Adapter represents a uniquely well-integrated SOA BPM Suite which in spite of its being open-source can compete with commercial offerings from big software vendors like Oracle, IBM and Progress Software, to name a few.

Major Features and Enhancements

This post is the first post in a series in which I will shortly describe and demonstrate the major features and enhancements that were done for our Intalio|BPM Adapter project. I intend to cover these topics in my next blog posts:

• Using existing services in your BPMN diagrams, I will show you how to comfortably browse through services that are available in your SOA infrastructure and reuse them in your BPMN diagrams using the drag and drop menu option. I will also show you how different communication styles are used and how your process react to changes in underlying services.
• Exposing BPMN processes as a service provider - in this part of the presentation I will describe and show you the two ways in which you can expose BPMN processes as a service and make it available for reuse in other processes and applications. I will also discuss how to deploy it and what communication styles you can use with it.

So stay tuned!

Renat Zubairov Development BPM, Intalio, SOA, SOPERA

SOA and Security

Security is a very important aspect of the SOA implementation, and one of the distinct features that distinguish an Enterprise-level product from the many ESB implementations you can find on the market today. There are many aspects of security that are relevant for any IT system implementation. Here are the most important items:

You can find more information about software security terminology in Wikipedia.

SOPERA ASF Provides Security Features

In SOPERA ASF, the services cover all of these security aspects. All you need to do is to take care of your business code and then simply annotate your providers with policies. Your business code remains “blissfully unaware” of the associated complexity behind the implementation! This is where SOPERA ASF Infrastructure does all the “behind-the-scenes” work.

Looking Under the Hood

In this blog I would like to show you how things work under the hood. Before we start, take a look at: encryption with asymmetric keys. In short each participant has a key-pair, for example, key A and B. You can use both for encryption and decryption, but once you encrypt something with one key you can only decrypt it with another matching key. For example, once you encrypt a message with key A you can only decrypt it with key B, and vice versa.

Encryption of Service Consumers & Providers

To apply these concepts in a distributed SOA environment we need an infrastructure and protocols for that, and that’s what SOPERA ASF does for you. Picture this: the service consumer and provider exchange messages and one of them (or both) would like to keep the messages encrypted (you can imagine that they are communicating via a public messaging service like Amazon SQS or any other channel that could be compromised). Both the provider and consumer have a key-pair (private and public keys), but they need to exchange the public keys before they can communicate. For example, when a service consumer sends a message to a service provider it should use the public key of the service provider to encrypt the message so that only the service provider (who has an appropriate private key) can decrypt it.

SOPERA ASF: Solving Problems

All these problems are what SOPERA ASF is solving for you. For the public key distribution we use services provided according to the XKMS specification, and this is how it works:

Example

Encryption sequence

• Consumer encrypts a message to a provider using the provider’s public key which it received from the XKMS. Since the consumer trust the XKMS, it can be sure that only the provider is able to decrypt and read message
• Provider decrypts the response using its private key. Later it encrypts the request using the consumer’s public key from the XKMS (we assume that the provider also trusts the XKMS here), so that only the consumer can decrypt it.
• Consumer decrypts the response using its private key.

In the end the business code of both the SOPERA Provider and Consumer remain unaware of all the complexity associated with encryption and the business code can be implemented without any additional effort and it remains blissfully oblivious as to whether it is communicating with an encrypted or unencrypted service provider or consumer.

Renat Zubairov Development

News Update
Lately, you may have seen in our blog a couple of posts on what we are working on currently in the BPM area. We are very excited about this work and its progress. However BPM is actually only part of the picture. It is only one part of what we proudly call our SOPERA SOA Suite. The other important part is Data Integration.
Data Integration
Data Integration was and it still is a very important aspect of any successful SOA implementation. Therefore, right now we are busy working on integrating one of the best open source products for data integration: Talend Integration Suite. Talend Integration Suite is an award winning data integration suite that makes data integration transparent and easy to accomplish. The extra bonus is that it’s also Eclipse-based, just like the rest of our tooling package!

Talend Open Studio has over 350 components that integrate with almost everything… starting from SAP and Salesforce integration, up to to commercial databases and different file formats. However, not everything is covered just yet. For example, in one of our customer cases we have noted a lack of Talend components for EDI support. At the same time we found a very nice implementation of an EDI parser (it’s probably better to say EDI-to-SAX transformer). So, we decided to contribute back to the community and initiated a new open source project.

SOPERADI-Smooks
Around two months ago we started an open source project. This project’s purpose is to create open source Talend components that are compatible with EDI data. Today, we are proud to present a beta-version of the project results, together with comprehensive documentation, and a couple of screencasts that show you how to use the new beta product.

On the project web site you will find more screencasts and documentation, as well as the ready-to-deploy component. So go ahead and use it in your projects today!

More information on SOPERA and EDI components you can find here: http://blog.sopera.com/2010/07/unedifact-support-overview/.

Renat Zubairov Development Data Integration, DI, SOPERA DI, Talend

Testing SOPERA-Intalio Add-On

In the last blog we shared with you our intention of combining SOA and BPM using SOPERA ASF and Intalio|BPM as the basis for an integration project. We now have some news about our approach to testing the SOPERA-Intalio add-on.

How we did testing
Previously, this is how we did testing; all our test cases were written, version-controlled, and deployed in our continuous build integration system (Cruise Control). For each possible Intalio|BPM scenario, one corresponding JUnit-based test was written. Each JUnit test case prepared test data and sent it to a specific Intalio process which in turn called up a SOPERA process. After that the JUnit testcase made an assertion on the Intalio response. Everything was then augmented with an automated Intalio/Intalio-SOPERA Add-On installation and process deployment so that we had a clean test-fixture each time.

What is new
After the initial implementation we realized that the only added value of JUnit in this case is assertion and test data setup, and as JUnit tests are sometimes hard to write and especially hard to maintain, we decided instead to simply use the full power of BPMN to move all our integration tests to a BPMN-only environment. We created a simple schema for process requests and responses where the request is left empty and the required response only requires a single boolean result.

We created a simple JUnit testcase which uses Apache ODE interfaces to discover processes and then one-by-one, verifies the response. For example, in the following picture you can see the integration test that tests the One-Way SOPERA call:

One-Way Integration Test

It is not easy to verify that the one-way service call was successful, so the BPMN Process above executes a one-way call (createLending) and then checks that the created service state is subsequently modified (by checking lending from the seekBook results).

All that the tester has to do now is:

1. Set up the BPM test environment.
2. Check out and modify the default test sets in the version control system.
3. Check it in, tag it, so that a new build is automatically created in cruise control: this starts the testing process.
4. Check the test results in Cruise Control.

That’s it!

Why we decided to take this approach

The following are the benefits we hope to achieve with this new approach:

1. All our development team members have the option of designing and testing a BPM process. We believe that using what we create is vital for successful product development.
2. Each of test cases created by a team member can be executed by any team member.
3. BPMN diagrams are much easier to understand and maintain.
4. It simulates the real user environment so that a form of in-house user acceptance testing is at play.
5. With the powerful BPMS Console from Intalio server we are presented with a perfect post-mortem examination process for our test cases and results. Since all our integration tests are BPMN processes we could investigate them step-by-step and find the reason for test failures.

Renat Zubairov Development BPM, SOA, SOPERA-Intalio, Testing