Hypothesis
The Orca2 LLM can be an effective Named Entity Recognition and Relation Extraction tool for aircraft system descriptions.
Setup
For this experiment, I am using ORCA2 7B GGUF model Q5_K_M. It tends to work quickly using the accelerate package and is recommended by the author for better fidelity and performance.
I decided to use Langchain to simplify use of the model so I can focus on the actual experiment.
Location
Github Location: ai4safety/MikeWorkspace/Goal1/TestNotebook.ipynb
Model Config
'context_length': 8000, 'max_new_tokens': 4096, 'gpu_layers': 32, 'temperature': 0.3, 'top_p': 0.45, 'top_k': 18, 'repetition_penalty': 1.1,
Low temperature is preferred as to not create any creative data.
top_p, top_k and repitition_penalty were set based on randomized trials over many runs.
context_length is set to an arbitrarily large number, this can affect performance, but so far so good.
max_new_tokens is set to the llama2 recommended value.
gpu_layers is with respect to what my RTX 3070Ti Mobile can handle without locking up.
System Prompt
The system prompt is created using langchain prompt templates, using two input variables, {input} and {json}, which will be the system description to be analyzed, and the example JSON output.
The system prompt tested for NER is:
You are an aircraft systems expert.
Perform the task of Named Entity Recognition (NER) on the input text.
You must format your output as a JSON vlaue that adheres to the example below.
Your output will be parsed and type-checked!
Example text:
The engine connects to the fuel pump.
The Avionics depends on the Electrical System.
Results in the json:
{json}
### Input:
{input}
### Response:
Valid JSON that adheres the requirements above and format of the example json output.
"""
The system prompt used for RE is:
You are an aircraft systems expert.
Perform the task of Relation Extraction (RE) on the input text.
You must format your output as a JSON vlaue that adheres to the example below.
Your output will be parsed and type-checked!
Example text:
The engine connects to the fuel pump.
The Avionics depends on the Electrical System.
Results in the json:
{json}
### Input:
{input}
### Response:
Valid JSON that adheres the requirements above and format of the example json output.
JSON Structure
# output json structures
example_NER_json = {
"entities": [
{
"name": "Engine",
"type": "SystemComponent",
},
{
"name": "Fuel Pump",
"type": "SystemComponent",
},
{
"name": "Avionics",
"type": "System",
},
{
"name": "Electrical System",
"type": "System",
}
]
}
example_RE_json = [
{
"entity1": "Engine",
"entity2": "Fuel Pump",
"relation_type": "Connects To"
},
{
"entity1": "Avionics",
"entity2": "Electrical System",
"relation_type": "Depends On"
}
]
Runs Using Data From Real System Descriptions
Source Data: https://openmbee.atlassian.net/l/cp/c2Qmroq2
# | Orca2 Output |
|---|---|
P1 | NER: {
"entities": [
{
"name": "Elevator",
"type": "SystemComponent"
},
{
"name": "Push-Pull-Rods",
"type": "SystemComponent"
},
{
"name": "Adjustable Stops",
"type": "SystemComponent"
},
{
"name": "Control Column",
"type": "SystemComponent"
},
{
"name": "Electrically Actuated Spring Trim",
"type": "System"
},
{
"name": "Spring-Loaded Switch",
"type": "System"
},
{
"name": "Trim Indicator",
"type": "System"
},
{
"name": "Resettable Circuit Breaker",
"type": "System"
}
]
}
RE: [{'entity1': 'Elevator', 'entity2': 'Push-Pull-Rods', 'relation_type': 'Controlled By'}, {'entity1': 'Adjustable Stops', 'entity2': 'Control Column', 'relation_type': 'Limit'}, {'entity1': 'Trim System', 'entity2': 'Spring Trim', 'relation_type': 'Actuated By'}, {'entity1': 'Trim System', 'entity2': 'Spring-Loaded Switch', 'relation_type': 'Controlled By'}, {'entity1': 'Trim Indicator', 'entity2': 'Instrument Panel', 'relation_type': 'Located On'}, {'entity1': 'Take-Off Position of Trim Indicator', 'entity2': 'Nose Down', 'relation_type': 'Marks'}]
|
P2 | NER: {
"entities": [
{
"name": "Elevator",
"type": "SystemComponent"
},
{
"name": "Yoke Tubes",
"type": "SystemComponent"
},
{
"name": "Cable Sector",
"type": "SystemComponent"
},
{
"name": "Torque Tube",
"type": "SystemComponent"
},
{
"name": "Push-Pull Linkage",
"type": "SystemComponent"
},
{
"name": "Elevator Bellcrank",
"type": "SystemComponent"
}
]
}
RE: [{'entity1': 'Elevator', 'entity2': 'Pitch Control', 'relation_type': 'Provides'}, {'entity1': 'Yoke Tubes', 'entity2': 'Motion Generation', 'relation_type': 'Generates Through'}, {'entity1': 'Push-pull Linkage', 'entity2': 'Cable Sector', 'relation_type': 'Connected To'}, {'entity1': 'Cable Sector', 'entity2': 'Torque Tube', 'relation_type': 'Mounted On'}, {'entity1': 'Forward Elevator Sector', 'entity2': 'Aft Elevator Sector', 'relation_type': 'Connected To'}, {'entity1': 'Aft Elevator Sector Pulley', 'entity2': 'Elevator Bellcrank', 'relation_type': 'Transmits Motion To'}]
|
P3 | NER: {
"entities": [
{
"name": "elevator",
"type": "SystemComponent"
},
{
"name": "yoke tubes",
"type": "SystemComponent"
},
{
"name": "push-pull linkage",
"type": "SystemComponent"
},
{
"name": "cable sector",
"type": "SystemComponent"
},
{
"name": "tailcone sector",
"type": "SystemComponent"
},
{
"name": "push/pull rod",
"type": "SystemComponent"
},
{
"name": "elevator sector",
"type": "SystemComponent"
},
{
"name": "elevator torque-tube",
"type": "SystemComponent"
}
]
}
RE: [{'entity1': 'Elevator', 'entity2': 'Aft Spar of Horizontal Stabilizer', 'relation_type': 'Attached To'}, {'entity1': 'Pilot’s Control Yokes', 'entity2': 'Elevator Control Sector', 'relation_type': 'Generates Motion Through'}]
|
P4 | NER: {
"entities": [
{
"name": "Landing Gear",
"type": "System"
},
{
"name": "Nose Gear",
"type": "SystemComponent"
},
{
"name": "Shock Absorber",
"type": "SystemComponent"
},
{
"name": "Main Landing Gear",
"type": "SystemComponent"
},
{
"name": "Rudder Pedals",
"type": "ControlSurface"
},
{
"name": "Spring Steel",
"type": "Material"
},
{
"name": "Hydraulically Actuated Disc Brakes",
"type": "SystemComponent"
},
{
"name": "5.00 x 5 Wheels",
"type": "SystemComponent"
},
{
"name": "AQUILA AT01 100",
"type": "AircraftModel"
},
{
"name": "Grass Surface",
"type": "SurfaceCondition"
},
{
"name": "Wheel Pants",
"type": "SystemComponent"
},
{
"name": "Mud Guards",
"type": "SystemComponent"
}
]
}
RE: [{'entity1': 'nose gear', 'entity2': 'rudder pedals', 'relation_type': 'Linked To'},
{'entity1': 'main landing gear struts', 'entity2': 'spring steel', 'relation_type': 'Made Of'},
{'entity1': 'main landing gear struts', 'entity2': 'touch-down loads', 'relation_type': 'Absorb'},
{'entity1': 'main landing gear', 'entity2': 'hydraulically actuated disc brakes', 'relation_type': 'Equipped With'}]
|
P5 | NER: {
"entities": [
{
"name": "main landing gear",
"type": "SystemComponent"
},
{
"name": "composite wing structure",
"type": "StructuralComponent"
},
{
"name": "wing spar",
"type": "StructuralComponent"
},
{
"name": "shear web",
"type": "StructuralComponent"
},
{
"name": "landing gear struts",
"type": "SystemComponent"
},
{
"name": "composite material",
"type": "Material"
},
{
"name": "fatigue resistance",
"type": "Property"
},
{
"name": "main wheels",
"type": "SystemComponent"
},
{
"name": "wheel plants",
"type": "SystemComponent"
},
{
"name": "bolt",
"type": "JoinMethod"
},
{
"name": "15 x 6.00 x 6 tire",
"type": "Tire"
},
{
"name": "innertube",
"type": "Component"
},
{
"name": "access plugs",
"type": "Feature"
},
{
"name": "tire inflation",
"type": "Action"
},
{
"name": "pressure checking",
"type": "Action"
},
{
"name": "brake",
"type": "SystemComponent"
},
{
"name": "independent",
"type": "Property"
},
{
"name": "hydraulically operated",
"type": "Property"
},
{
"name": "single-disc type",
"type": "Property"
}
]
}
RE: [{'entity1': 'main landing gear', 'entity2': 'composite wing structure', 'relation_type': 'bolted to'}, {'entity1': 'landing gear struts', 'entity2': 'composite material', 'relation_type': 'constructed of'}]
|
P6 | NER: {
"entities": [
{
"name": "Landing Gear",
"type": "SystemComponent"
},
{
"name": "Nose Wheel",
"type": "SystemComponent"
},
{
"name": "Main Wheels",
"type": "SystemComponent"
},
{
"name": "Tubular Spring Steel Main Landing Gear Struts",
"type": "SystemComponent"
},
{
"name": "Steel Cross Tubes",
"type": "SystemComponent"
},
{
"name": "Nose Gear Shock Strut",
"type": "SystemComponent"
},
{
"name": "Tires",
"type": "SystemComponent"
}
]
}
RE: [{'entity1': 'landing gear', 'entity2': 'nose wheel', 'relation_type': 'Connects To'},
{'entity1': 'landing gear', 'entity2': 'main wheels', 'relation_type': 'Connects To'},
{'entity1': 'landing gear', 'entity2': 'tubular spring steel main landing gear struts', 'relation_type': 'Made Of'},
{'entity1': 'landing geyer', 'entity2': 'steel cross tubes', 'relation_type': 'Made Of'},
{'entity1': 'landing gear', 'entity2': 'nose gear shock strut', 'relation_type': 'Made Of'},
{'entity1': 'nose gear shock strut', 'entity2': 'air and oil-filled', 'relation_type': 'Filled With'},
{'entity1': 'landing gear', 'entity2': 'tubular spring steel main landing gear struts', 'relation_type': 'Interacts With'},
{'entity1': 'tubular spring steel main landing gear struts', 'entity2': 'steel cross tubes', 'relation_type': 'Interacts With'},
{'entity1': 'landing gear', 'entity2': 'oversize tires', 'relation_type': 'Optionally Installed For'},
{'entity1': 'oversize tires', 'entity2': 'propeller ground clearance', 'relation_type': 'Provides'},
{'entity1': 'oversize tires', 'entity2': 'footprint', 'relation_type': 'Provides'}]
|
P7 | NER: {
"entities": [
{
"name": "Fuel Tank Selector",
"type": "SystemComponent"
},
{
"name": "LEFT",
"type": "Label"
},
{
"name": "RIGHT",
"type": "Label"
},
{
"name": "ON",
"type": "PositionControl"
},
{
"name": "OFF",
"type": "PositionControl"
}
]
}
RE: [{'entity1': 'Fuel Tank Selector', 'entity2': 'Wing Tank Shutoff Valve', 'relation_type': 'Control By'}, {'entity1': 'LEFT', 'entity2': 'RIGHT', 'relation_type': 'Position'}]
|
P8 | NER: {
"entities": [
{"name": "Fuel Selector Valve", "type": "System"},
{"name": "Selector Handle", "type":": "SystemComponent"},
{"name": "Fuel Tank", "type": "System"}
]
}
RE: [{'entity1': 'Fuel Selector Valve', 'entity2': 'Control Panel', 'relation_type': 'Mounted On'}, {'entity1': 'Selector Handle', 'entity2': 'Position', 'relation_type': 'Has'}]
|
P9 | NER: {
"entities": [
{
"name": "Fuel Selector Valve",
"type": "SystemComponent"
},
{
"name": "Left Tank",
"type": "FuelTank"
},
{
"name": "Right Tank",
"type": "FuelTank"
},
{
"name": "Fuel Indicator",
"type": "SystemComponent"
},
{
"name": "Fuel Selector Knob Release",
"type": "SystemComponent"
}
]
}
RE: [{'entity1': 'Fuel Selector Valve', 'entity2': 'LEFT Tank', 'relation_type': 'Allows Fuel Flow From'}, {'entity1': 'Fuel Selector Valve', 'entity2': 'RIGHT Tank', 'relation_type': 'Allows Fuel Flow From'}, {'entity1': 'Fuel Selector Valve', 'entity2': 'OFF', 'relation_type': 'Cuts Off Fuel Flow From Both Tanks'}, {'entity1': 'LEFT Tank', 'entity2': 'RIGHT Tank', 'relation_type': 'Compares To'}, {'entity1': 'Fuel Selector Valve', 'entity2': 'OFF', 'relation_type': 'Requires Raising Fuel Selector Knob Release And Rotating The Knob Of OFF'}]
|
P10 | NER: {
"entities": [
{
"name": "Fuselage",
"type": "BodyPart"
},
{
"name": "Vertical Stabilizer",
"type": "BodyPart"
},
{
"name": "Firewall",
"type": "SystemComponent"
},
{
"name": "Ceramic Fleece",
"type": "Material"
},
{
"name": "Stainless Steel Sheet",
"type": "Material"
},
{
"name": "Landing Gear Frame",
"type": "SystemComponent"
},
{
"name": "Seat Frame",
"type": "SystemComponent"
},
{
"name": "Roll Cage",
"type": "SystemComponent"
}
]
}
RE: [{'entity1': 'Fuselage', 'entity2': 'Vertical Stabilizer', 'relation_type': 'In Two Half Shells'},
{'entity1': 'Fuselage', 'entity2': 'Shell', 'relation_type': 'Made Of'},
{'entity1': 'Firewall', 'entity2': 'Engine', 'relation_type': 'Covers'},
{'entity1': 'Firewall', 'entity2': 'Fire-Resistant Ceramic Fleece', 'relation_type': 'Covered With'},
{'entity1': 'Firewall', 'entity2': 'Stainless Steel Sheet', 'relation_type': 'Covered With'},
{'entity1': 'Landing Gear Frame', 'entity2': 'Seat Frame', 'relation_type': 'Supports'},
{'entity1': 'Frame', 'entity2': 'Roll Cage', 'relation_type': 'Forms'}
|