Darwin4Trains
Contents
About
Darwin 4 Trains (D4T) is a data service provided by the Rail Delivery Group containing real time train centric data about passenger services on the UK rail network. D4T provides data in a JSON format via RESTful webservices and a publish subscribe model using Apache Active MQ.
The data provided by D4T is intended for use by Train Operating Companies (TOCs) within their passenger information systems (PIS) on board trains as well as by public application developers via the Open Rail Data Portal. D4T sources data from Darwin, Knowledge Base and Transport for London in real time then pivots it to put it in the context of a train greatly simplifying the presentation of train data.
Data
Darwin 4 Trains presents three types of messages via AMQ
Journey Status
The Journey Status message is the most commonly published message in Darwin 4 Trains. Each message is the current state of a service and shows all data about that service for calling points that have not yet been passed. The Journey Status message shows key identifiers for that service including the RID and SSD/UID along with the calling points, data on any planned or unplanned disruptions as well the current schedule for the journey.
A sample message is available here AMQ Train Message
Station
Station messages show key information about the facilities available at a station. A message is published once per day for each CRS code currently used on the network and station messages are not filtered by the AMQ selectors. There are three types of data within each station message:
- Station Facilities
- Accessibility
- Onward Travel data such as bus routes and other third party travel services (only available in England for stations NOT run by Network Rail)
A sample message is available here: AMQ Station Facilities Message
TFL
TFL (Transport for London) messages contain the TFL rainbow showing the status of each TFL train service on the network. These messgaes are published each time the TFL rainbow is updated and always show the whole rainbow
A sample message is available here: AMQ Transport For London
Snapshot
In the instance where you first connect to Darwin4Trains, if you require the current state, you will need to request a snapshot. Example snapshot messages:
First Message: Snapshot Start
Last Message: Snapshot End
TFL Message: AMQ Transport For London
Station Facility Message: AMQ Station Facilities Message
Train AMQ Message: AMQ Train Message
How do I get Real-time Data
Real time data is published via an Apache Active MQ end point. Data is near real time with a target of being no more then 400 ms behind the source Darwin system.
Connecting to AMQ
For more information on requesting an account To connect to the AMQ queue please request a unique Client ID from your representative at the Rail Delivery group.
Once received you will need to create a new connection to "activemq:failover:(tcp://34.251.253.146:61616,tcp://34.250.11.236:61616)?randomize=false" using your provided username and password. Then consume the messages on the "Darwin4Trains" topic. If you wish to filter, please use a selector when creating your consumer. Example filters are available below.
Below are a number of examples showing how to connect to the D4T Active MQ and how to apply filters to personalise your view of the feed.
JMS Filtering
AMQ Filtering is accomplished using JMS selectors. Additional information on how to use these selectors: [1] [2]
Filterable Properties
| Train | TFL | Station Facilities |
|---|---|---|
|
|
|
Filter Elements
| Element | Description |
|---|---|
| AND, OR, NOT | Logical operators |
| =, <>, <, <=, >, >= | Comparison operators |
| LIKE | String comparison with wildcards '_' and '%' |
| IN | Find value in set of strings |
Example Filters
Unfiltered, To disable the filter, change the IsSelectorEnabled to false
<add key="IsSelectorEnabled" value="false" />
Only return trains associated with NT
<add key="SelectorFilter" value="AtocCode = 'NT'" />
Only return Station messages
<add key="SelectorFilter" value="Type = 'StationFacilitiesViewModel'" />
Only return TFL messages
<add key="SelectorFilter" value="Type = 'TflAmqViewModel'" />
Return Trains passing through LST and TFL messages
<add key="SelectorFilter" value="CRS = 'LST' OR Type = 'TflAmqViewModel'" />
Requesting a Snapshot
To request a snapshot using your own AMQ client:
- Pause your D4T feed
- Create Temporary Queue
- Create a Consumer on the queue, your snapshot response will be sent to this queue.
- Send the filter config in a message to the queue. You should set NMSType on your message to "snapshot"
- Receive the Snapshot Start Message from the snapshot
- Process all messages until an End Message is received
- Read the Timestamp from the Snapshot End message
- Resume the D4T feed and discard any messages that were sent before the Timestamp contained in the Snapshot End message
Filter Config
To filter the snapshot then after creating the consumer place a message on the queue containing your filters in the format below. This example returns a snapshot for all trains that call at Ipswich station
"{ 'crs':['IPS'], 'toc':[], 'rid':[], 'uidssd':[] }"
Each item is an array so multiple values can be entered as follows:
- crs : an array of three-char CRS codes
- toc : an array of two-char TOC (region) codes
- rid : an array of 14-digit train RIDs
- uidssd : array of UID and SSD combinations in the format "Xnnnnn yyyy-MM-dd"
Multiple filters can be specified, these (per spec) are additive filters so the following will return all trains stopping at Ipswich, Bridlington and with the specified UID SSD combination
"{ 'crs':['IPS','BDT'], 'toc':[], 'rid':[], 'uidssd':['G21635 2017-08-01'] }"
Checking for and Handling Errors
This section describes what to do as a consumer in the case of errors in the Darwin 4 Trains data feed or in the case that the consumer has an issue.
Message Confirmation
Due to the nature of the AMQ virtual queues in place, the order in which a message is placed on that queue is the order in which it will be consumed. To ensure delivery of a message AMQ expects to receive an acknowledgement of receipt. If the message does not send the ACK then it is returned to the front of the queue to be attempted again. Auto Acknowledgement of messages in AMQ should not be used with D4T. The AMQ client should be configured to use Manual Acknowledgement only.
Message Loss
All D4T messages delivered via AMQ have a global sequence number in the message header within the field "Sequence." This is a unique sequence number specific to D4T AMQ messages and it is comprised YY + Day of Year + an 7 digit incrementing sequence number. E.g. 1711100000012. The message sequence number is incremented for every message produced by D4T and messages are always added to the outbound queue in sequence order. The consumer should always check that the current message they are processing has a higher sequence number than the last message they received. If the sequence number is lower than or equal to the last sequence number then it is recommended that the consumer request a snapshot before continuing to process messages.
Client error scenarios
| Scenario | Action | Mitigation |
|---|---|---|
| Client loses connection to Darwin 4 Trains AMQ for less than 10 minutes | Release the lock held on the active message after a set period (30 seconds). This message is then placed at the front of the queue from the client to consume again | All messages will continue to accrue for up to 10 minutes at which moment the client’s subscription will be reset.
If the consumer becomes unsure about the state of their data they can request a snapshot from Darwin 4 Trains |
| Client loses connection to Darwin 4 Trains AMQ for more than 10 minutes | The client subscription will be removed. Upon reconnection, the user will get a new subscription and will have to request a new snapshot. | As the subscription, has been removed the client will have to create a new subscription which involves requesting a snapshot. |
| Client crashes or fails to acknowledge receipt of message from Darwin for trains AMQ | Release the lock held on the active message after a set period (30 seconds). This message is then placed at the front of the queue from the client to consume again. | All messages will continue to accrue for up to 10 minutes at which moment the client’s subscription will be reset.
Each message sent to AMQ from D4T includes a unique message Id (MNSMessageId), if required a consumer can use this to verify they have not processed the message before. If the consumer becomes unsure about the state of their data, they can request a snapshot from Darwin 4 Trains |
Getting Non Real-time data via the REST API
Access Controls
In order to successfully get a response to any of the API calls, users need to sign their requests using Sig4 (http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html) and the secret/access key in the XLSX supplied. The requests will need to be signed with the region as eu-west-1 (case sensitive) and service name as execute-api.
This can be easily achieved via a program called Postman (download [[3]]). Once you have this installed, you will need to enter your configuration, example below
Postman example configuration [[4]]
![[4]](https://ijyi.blob.core.windows.net/ijyiwiki/Postman.PNG){kind=link}
| Url Example (Ipswich Arrivals) | https://api.prod01.d4t-env.nationalrail.co.uk/v1/arrivals/ips |
| AccessKey | From supplied XLSX |
| SecretKey | From supplied XLSX |
| AWS Region | eu-west-1 |
| Service Name | execute-api |
Available Operations
| Operation Name | Url | Description | Further Information |
|---|---|---|---|
| Arrivals | GET /v1/arrivals/{crs} | Return a collection of arrivals based on the computer reservation system code of a station. | Arrivals by CRS |
| Departures | GET /v1/departuresfiltered/{crs} | Returns Departures from the first station to another station and/or from a specific datetime. | Departures Filtered by CRS |
| Departures | GET /v1/departures/{crs} | Return a collection of Departures based on the computer reservation system code of a station. | Departures by CRS |
| Journey Status | GET /v1/journeystatusFromRid/{rid} | Returns the journey status of a service based on the rid. | Journey Status from RID |
| Journey Status | GET /v1/journeystatusFromUidSsd/{uid}/{ssd} | Returns the journey status of a service from a specified unique identifier and schedule start date. | Journey Status from UID SSD |
| Onward Travel | GET /v1/onwardtravel | Return a collection of onward travel data based on the computer reservation system code of a station. | Onward Travel |
| Reference Data | GET /v1/referencedata | Returns all reference data for use in Darwin4Trains. | Reference Data |
| Stations | GET /v1/stationFacilitiesFromCrs/{crs} | Returns station facilities at a specified location (CRS). | Station Facilities |
| Underground | GET /v1/underground | Get the TFL rainbow. | Underground |
Examples
Code examples for connecting to the AMQ and requesting a snapshot are available in Github.
Support
In the case of a support issue please either contact your representative at the Rail Delivery Group or in the case of emergencies contact the support help desk by emailing service.desk@nationalrail.co.uk or phoning 08453661261