Logistics firms in search of Big Data may gain more by digging deeper for Small Data

© Libux77

Santiago, the main character in Paul Coelho’s bestseller The Alchemist, went on a long journey in search of a treasure. Only to find that it was buried close to home – right below the place where he used to sleep.

This modern day fable might contain a valuable lesson for companies in search of Big Data.

Numerous articles are being published on the potential of Big Data in logistics, but those that chart its actual, successful use are far more scarce. It is still early days, but in my view we have yet to pass through the Trough of Disillusionment before we hit the Slope of Enlightenment (see Gartner’s Hype Cycle stages).

However, this does not mean that there are not data treasures to be found right under our feet. We just need to dig a bit deeper.

Every day, countless manual and electronic transactions are conducted in the core systems of airlines and forwarders alike. Each of these transactions is a single pixel of a larger picture: eg, the current state of a specific flight or consolidation.

I like to refer to this source of information as “Small Data”. It is Small Data as it contains only internal information and covers just a portion of the possible sources Big Data explorers are considering.

The main advantage of Small Data over Big Data is that most of its data elements follow a strict (IATA) data format. This standardisation greatly reduces the complexity of interpreting these individual transactions, which is one of largest hurdles when dealing with Big Data.

Three obvious message types which contain this kind of Small Data that could be put to great use are FSU (updates), FFM (manifests) and FWB (bookings) messages.

The next step is to interpret these messages as a collective – preferably several times a day – and thus follow the development of a single flight or consolidation from cradle to grave.

This collective interpretation needs to be a hybrid approach, to be successful.

Algorithms firstly conduct the leg work and provide their recommendations via a simple user interface. It is then up to the cargo professional to make the final assessment and take the appropriate decision. For example, increasing the hurdle rate for a specific flight; or moving a house air waybill from one consolidation to the next in order to achieve a higher ‘volume kill’.

By continuously gathering and interpreting these different data elements, airlines and forwarders will be able to ‘see’ quicker what is really happening than they currently can.

This hybrid approach to using Small Data is a treasure waiting to be found. I will not be surprised if more companies will start digging shortly. The companies who find it first will gain more, and suffer less, from the sudden market changes that are likely to feature in the near future.

Niall van de Wouw,  managing director of CLIVE, will be speaking on this topic during the Technology Track at the upcoming World Cargo Symposium in Berlin. He can be reached at [email protected]