ORCA2 NER/RE Experiment

By Michael Mustillo
6 min

Completes task: https://openmbee.atlassian.net/browse/AI4-20

Hypothesis

The Orca2 LLM can be an effective Named Entity Recognition and Relation Extraction tool for aircraft system descriptions.

Setup

For this experiment, I am using ORCA2 7B GGUF model Q5_K_M. It tends to work quickly using the accelerate package and is recommended by the author for better fidelity and performance.

I decided to use Langchain to simplify use of the model so I can focus on the actual experiment.

Location

Github Location: ai4safety/MikeWorkspace/Goal1/TestNotebook.ipynb

Model Config

'context_length': 8000, 'max_new_tokens': 4096, 'gpu_layers': 32, 'temperature': 0.3, 'top_p': 0.45, 'top_k': 18, 'repetition_penalty': 1.1,

  • Low temperature is preferred as to not create any creative data.

  • top_p, top_k and repitition_penalty were set based on randomized trials over many runs.

  • context_length is set to an arbitrarily large number, this can affect performance, but so far so good.

  • max_new_tokens is set to the llama2 recommended value.

  • gpu_layers is with respect to what my RTX 3070Ti Mobile can handle without locking up.

System Prompt

The system prompt is created using langchain prompt templates, using two input variables, {input} and {json}, which will be the system description to be analyzed, and the example JSON output.

The system prompt tested for NER is:

You are an aircraft systems expert. Perform the task of Named Entity Recognition (NER) on the input text. You must format your output as a JSON vlaue that adheres to the example below. Your output will be parsed and type-checked! Example text: The engine connects to the fuel pump. The Avionics depends on the Electrical System. Results in the json: {json} ### Input: {input} ### Response: Valid JSON that adheres the requirements above and format of the example json output. """

The system prompt used for RE is:

You are an aircraft systems expert. Perform the task of Relation Extraction (RE) on the input text. You must format your output as a JSON vlaue that adheres to the example below. Your output will be parsed and type-checked! Example text: The engine connects to the fuel pump. The Avionics depends on the Electrical System. Results in the json: {json} ### Input: {input} ### Response: Valid JSON that adheres the requirements above and format of the example json output.

 

JSON Structure

# output json structures example_NER_json = {     "entities": [         {             "name": "Engine",             "type": "SystemComponent",         },         {             "name": "Fuel Pump",             "type": "SystemComponent",         },         {             "name": "Avionics",             "type": "System",         },         {             "name": "Electrical System",             "type": "System",         }     ] } example_RE_json = [     {         "entity1": "Engine",         "entity2": "Fuel Pump",         "relation_type": "Connects To"     },     {         "entity1": "Avionics",         "entity2": "Electrical System",         "relation_type": "Depends On"     } ]

 

Runs Using Data From Real System Descriptions

Source Data: https://openmbee.atlassian.net/l/cp/c2Qmroq2

#

Orca2 Output

#

Orca2 Output

P1

NER:

{ "entities": [ { "name": "Elevator", "type": "SystemComponent" }, { "name": "Push-Pull-Rods", "type": "SystemComponent" }, { "name": "Adjustable Stops", "type": "SystemComponent" }, { "name": "Control Column", "type": "SystemComponent" }, { "name": "Electrically Actuated Spring Trim", "type": "System" }, { "name": "Spring-Loaded Switch", "type": "System" }, { "name": "Trim Indicator", "type": "System" }, { "name": "Resettable Circuit Breaker", "type": "System" } ] }

RE:

[ {'entity1': 'Elevator', 'entity2': 'Push-Pull-Rods', 'relation_type': 'Controlled By'}, {'entity1': 'Adjustable Stops', 'entity2': 'Control Column', 'relation_type': 'Limit'}, {'entity1': 'Trim System', 'entity2': 'Spring Trim', 'relation_type': 'Actuated By'}, {'entity1': 'Trim System', 'entity2': 'Spring-Loaded Switch', 'relation_type': 'Controlled By'}, {'entity1': 'Trim Indicator', 'entity2': 'Instrument Panel', 'relation_type': 'Located On'}, {'entity1': 'Take-Off Position of Trim Indicator', 'entity2': 'Nose Down', 'relation_type': 'Marks'} ]

P2

NER:

{ "entities": [ { "name": "Elevator", "type": "SystemComponent" }, { "name": "Yoke Tubes", "type": "SystemComponent" }, { "name": "Cable Sector", "type": "SystemComponent" }, { "name": "Torque Tube", "type": "SystemComponent" }, { "name": "Push-Pull Linkage", "type": "SystemComponent" }, { "name": "Elevator Bellcrank", "type": "SystemComponent" } ] }

RE:

[ {'entity1': 'Elevator', 'entity2': 'Pitch Control', 'relation_type': 'Provides'}, {'entity1': 'Yoke Tubes', 'entity2': 'Motion Generation', 'relation_type': 'Generates Through'}, {'entity1': 'Push-pull Linkage', 'entity2': 'Cable Sector', 'relation_type': 'Connected To'}, {'entity1': 'Cable Sector', 'entity2': 'Torque Tube', 'relation_type': 'Mounted On'}, {'entity1': 'Forward Elevator Sector', 'entity2': 'Aft Elevator Sector', 'relation_type': 'Connected To'}, {'entity1': 'Aft Elevator Sector Pulley', 'entity2': 'Elevator Bellcrank', 'relation_type': 'Transmits Motion To'} ]

P3

NER:

{ "entities": [ { "name": "elevator", "type": "SystemComponent" }, { "name": "yoke tubes", "type": "SystemComponent" }, { "name": "push-pull linkage", "type": "SystemComponent" }, { "name": "cable sector", "type": "SystemComponent" }, { "name": "tailcone sector", "type": "SystemComponent" }, { "name": "push/pull rod", "type": "SystemComponent" }, { "name": "elevator sector", "type": "SystemComponent" }, { "name": "elevator torque-tube", "type": "SystemComponent" } ] }

RE:

[ {'entity1': 'Elevator', 'entity2': 'Aft Spar of Horizontal Stabilizer', 'relation_type': 'Attached To'}, {'entity1': 'Pilot’s Control Yokes', 'entity2': 'Elevator Control Sector', 'relation_type': 'Generates Motion Through'} ]

P4

NER:

{ "entities": [ { "name": "Landing Gear", "type": "System" }, { "name": "Nose Gear", "type": "SystemComponent" }, { "name": "Shock Absorber", "type": "SystemComponent" }, { "name": "Main Landing Gear", "type": "SystemComponent" }, { "name": "Rudder Pedals", "type": "ControlSurface" }, { "name": "Spring Steel", "type": "Material" }, { "name": "Hydraulically Actuated Disc Brakes", "type": "SystemComponent" }, { "name": "5.00 x 5 Wheels", "type": "SystemComponent" }, { "name": "AQUILA AT01 100", "type": "AircraftModel" }, { "name": "Grass Surface", "type": "SurfaceCondition" }, { "name": "Wheel Pants", "type": "SystemComponent" }, { "name": "Mud Guards", "type": "SystemComponent" } ] }

RE:

[ {'entity1': 'nose gear', 'entity2': 'rudder pedals', 'relation_type': 'Linked To'}, {'entity1': 'main landing gear struts', 'entity2': 'spring steel', 'relation_type': 'Made Of'}, {'entity1': 'main landing gear struts', 'entity2': 'touch-down loads', 'relation_type': 'Absorb'}, {'entity1': 'main landing gear', 'entity2': 'hydraulically actuated disc brakes', 'relation_type': 'Equipped With'} ]

P5

NER:

{ "entities": [ { "name": "main landing gear", "type": "SystemComponent" }, { "name": "composite wing structure", "type": "StructuralComponent" }, { "name": "wing spar", "type": "StructuralComponent" }, { "name": "shear web", "type": "StructuralComponent" }, { "name": "landing gear struts", "type": "SystemComponent" }, { "name": "composite material", "type": "Material" }, { "name": "fatigue resistance", "type": "Property" }, { "name": "main wheels", "type": "SystemComponent" }, { "name": "wheel plants", "type": "SystemComponent" }, { "name": "bolt", "type": "JoinMethod" }, { "name": "15 x 6.00 x 6 tire", "type": "Tire" }, { "name": "innertube", "type": "Component" }, { "name": "access plugs", "type": "Feature" }, { "name": "tire inflation", "type": "Action" }, { "name": "pressure checking", "type": "Action" }, { "name": "brake", "type": "SystemComponent" }, { "name": "independent", "type": "Property" }, { "name": "hydraulically operated", "type": "Property" }, { "name": "single-disc type", "type": "Property" } ] }

RE:

[ {'entity1': 'main landing gear', 'entity2': 'composite wing structure', 'relation_type': 'bolted to'}, {'entity1': 'landing gear struts', 'entity2': 'composite material', 'relation_type': 'constructed of'} ]

P6

NER:

{ "entities": [ { "name": "Landing Gear", "type": "SystemComponent" }, { "name": "Nose Wheel", "type": "SystemComponent" }, { "name": "Main Wheels", "type": "SystemComponent" }, { "name": "Tubular Spring Steel Main Landing Gear Struts", "type": "SystemComponent" }, { "name": "Steel Cross Tubes", "type": "SystemComponent" }, { "name": "Nose Gear Shock Strut", "type": "SystemComponent" }, { "name": "Tires", "type": "SystemComponent" } ] }

RE:

[ {'entity1': 'landing gear', 'entity2': 'nose wheel', 'relation_type': 'Connects To'}, {'entity1': 'landing gear', 'entity2': 'main wheels', 'relation_type': 'Connects To'}, {'entity1': 'landing gear', 'entity2': 'tubular spring steel main landing gear struts', 'relation_type': 'Made Of'}, {'entity1': 'landing geyer', 'entity2': 'steel cross tubes', 'relation_type': 'Made Of'}, {'entity1': 'landing gear', 'entity2': 'nose gear shock strut', 'relation_type': 'Made Of'}, {'entity1': 'nose gear shock strut', 'entity2': 'air and oil-filled', 'relation_type': 'Filled With'}, {'entity1': 'landing gear', 'entity2': 'tubular spring steel main landing gear struts', 'relation_type': 'Interacts With'}, {'entity1': 'tubular spring steel main landing gear struts', 'entity2': 'steel cross tubes', 'relation_type': 'Interacts With'}, {'entity1': 'landing gear', 'entity2': 'oversize tires', 'relation_type': 'Optionally Installed For'}, {'entity1': 'oversize tires', 'entity2': 'propeller ground clearance', 'relation_type': 'Provides'}, {'entity1': 'oversize tires', 'entity2': 'footprint', 'relation_type': 'Provides'} ]

P7

NER:

{ "entities": [ { "name": "Fuel Tank Selector", "type": "SystemComponent" }, { "name": "LEFT", "type": "Label" }, { "name": "RIGHT", "type": "Label" }, { "name": "ON", "type": "PositionControl" }, { "name": "OFF", "type": "PositionControl" } ] }

RE:

[ {'entity1': 'Fuel Tank Selector', 'entity2': 'Wing Tank Shutoff Valve', 'relation_type': 'Control By'}, {'entity1': 'LEFT', 'entity2': 'RIGHT', 'relation_type': 'Position'} ]

P8

NER:

{ "entities": [ {"name": "Fuel Selector Valve", "type": "System"}, {"name": "Selector Handle", "type":": "SystemComponent"}, {"name": "Fuel Tank", "type": "System"} ] }

RE:

[ {'entity1': 'Fuel Tank Selector', 'entity2': 'Wing Tank Shutoff Valve', 'relation_type': 'Control By'}, {'entity1': 'LEFT', 'entity2': 'RIGHT', 'relation_type': 'Position'} ]

P9

NER:

{ "entities": [ { "name": "Fuel Selector Valve", "type": "SystemComponent" }, { "name": "Left Tank", "type": "FuelTank" }, { "name": "Right Tank", "type": "FuelTank" }, { "name": "Fuel Indicator", "type": "SystemComponent" }, { "name": "Fuel Selector Knob Release", "type": "SystemComponent" } ] }

RE:

[ {'entity1': 'Fuel Selector Valve', 'entity2': 'LEFT Tank', 'relation_type': 'Allows Fuel Flow From'}, {'entity1': 'Fuel Selector Valve', 'entity2': 'RIGHT Tank', 'relation_type': 'Allows Fuel Flow From'}, {'entity1': 'Fuel Selector Valve', 'entity2': 'OFF', 'relation_type': 'Cuts Off Fuel Flow From Both Tanks'}, {'entity1': 'LEFT Tank', 'entity2': 'RIGHT Tank', 'relation_type': 'Compares To'}, {'entity1': 'Fuel Selector Valve', 'entity2': 'OFF', 'relation_type': 'Requires Raising Fuel Selector Knob Release And Rotating The Knob Of OFF'} ]

P10

NER:

{ "entities": [ { "name": "Fuselage", "type": "BodyPart" }, { "name": "Vertical Stabilizer", "type": "BodyPart" }, { "name": "Firewall", "type": "SystemComponent" }, { "name": "Ceramic Fleece", "type": "Material" }, { "name": "Stainless Steel Sheet", "type": "Material" }, { "name": "Landing Gear Frame", "type": "SystemComponent" }, { "name": "Seat Frame", "type": "SystemComponent" }, { "name": "Roll Cage", "type": "SystemComponent" } ] }

RE:

[ {'entity1': 'Fuselage', 'entity2': 'Vertical Stabilizer', 'relation_type': 'In Two Half Shells'}, {'entity1': 'Fuselage', 'entity2': 'Shell', 'relation_type': 'Made Of'}, {'entity1': 'Firewall', 'entity2': 'Engine', 'relation_type': 'Covers'}, {'entity1': 'Firewall', 'entity2': 'Fire-Resistant Ceramic Fleece', 'relation_type': 'Covered With'}, {'entity1': 'Firewall', 'entity2': 'Stainless Steel Sheet', 'relation_type': 'Covered With'}, {'entity1': 'Landing Gear Frame', 'entity2': 'Seat Frame', 'relation_type': 'Supports'}, {'entity1': 'Frame', 'entity2': 'Roll Cage', 'relation_type': 'Forms'} ]

Related content