SimPy Cyber RL Env

Components

Forwarder Device

A router is a networking device used for forwarding data packets between two networks. It functions in the network layer of the OSI layer. Conventionally, the router software has two functional processing units: a) Control Plane b) Forwarding Plane. In the control plane, the router maintains a routing table that maintains which path would be used to forward a packet and along which interface. This routing table is statically configured or updated dynamically using dynamic routing protocols such as OSPF, EIGRP, RIP, etc. The time taken to forward packets depends on the processing time of parsing the packet, and searching the next hop information from the routing table. Hence, the service rate, as well as, the queue limit for every router is modeled. The queue limit indicate the amount of byte the router can forward at a given instance. In the forwarding plane, the router simply forwards the packet to the desired interface based on the routing rule. In the MDP model, modelling the router drop rate as state, would play a crucial role in enforcing new routes. The router packet drop rate is defined as ratio of the number of dropped packets and the received packets.

Channel

The channel models the latency and bandwidth between each node in the network model. Based on the traffic, the channels update it utilization rate and computes the available channel capacity with the update frequency in the Simpy DES. A packet injected into the channel is dropped if the total bytes in channel crosses the available channel bandwith. For the evaluation of the environment, the experiments are performed under varying channel bandwidths.

Data Concentrator (DC)

It is acting as data collector and forwarder for all the smart meters in the zones. When a fault occurs in the distribution feeder, the relay captures it and forward it to the Data Concentrator (DC). The DC within the zone forwards the state to the Distribution System Operator (DSO) which acts as the Data Aggregator. In the simulation model, it is assumed all the sensor data are accumulated at the DC, hence the payload size of the packet sent to the DA depends on the number of components i.e. transmission lines, buses, switches, etc within a zone. Conventionally, the DC communicates with sensors using power line carriers, while they communicate to DSO through the Wide Area Network (WAN).

Distribution System Operator (DSO) / Data Aggregator (DA)

Packets are received from the DC from different zones in the system. This node acts as the DSO collecting data from each DC. In the MDP model, the goal state to be the state when the data aggregator accept atleast Ng traffic within a timeframe from the DCs of each zone.

Classes and Functions

Created on Fri July 1 15:24:05 2022

@author: abhijeetsahu

This environment is created for episodic interaction of the Simpy based Communication simulator for training RL agent, similar to an Open AI Gym environment with implementation of the channel model

class CyberWithChannelEnvSB_123_Experimentation.BaseEnv(deviceCount: int)

Bases: gym.core.Env

ASSIGNMENT_DURATION_FACTOR = 1000
MAX_RECEIVER_DEGREE = 5
metadata = {'render.modes': ['human']}
render(mode='human', close=False)

Renders the environment to stdout.

seed(seed=None)

Sets the seed for this environment’s random number generator and returns it in a single-item list.

class CyberWithChannelEnvSB_123_Experimentation.CyberEnv(provided_graph=None, channelModel=True, envDebug=True, R2_qlimit=100, ch_bw=1000, with_threat=False, comp_zones=None)

Bases: CyberWithChannelEnvSB_123_Experimentation.BaseEnv

This is the Simpy based cyber environment class

class Channel(env, id, src, dest, bw=2000, delay=1, limit_bytes=True, debug=False, snr=10)

Bases: object

This is the Channel class that models the communication channel. the overall delay depends on multiple factors : transmission delay + propagation delay + queueing delay + processing delay the transmission delay depends on the channel capacity the propagation delay : distance/speed of transmitting medium such as light or sound the queueing delay is the wait time and processing at every switch and router while the processing delay is all the time consumed in encapsulation and decapsulation of headers + fragmentation and reassembly etc etc

In our simulation, we implement dynamic update on channel_capacity that changes the channel capacity which changes the transmission delay and keep the propagation delay fixed

put(pkt)

Forwards the packet into the next router and add transmission delay

run()
update_ur()

This function updates the channel utilization rate

class ForwarderDevice(env, id, rate, qlimit=None, limit_bytes=True, debug=False)

Bases: envs.simpy_env.SimComponentsDynamic.SwitchPort

This is the ForwarderDevice class acting as the router

change_receiver(new_receiver)

This function re-routes or update the routing table in the forwarder

Parameters

new_receiver (str) – the next hop router selected

forwarderProcess()

This function forwards the packet i.e. receives packet from incoming channel and forwards to the out-going channel Here it also checks the queue limit and takes the processing time.

update_router_queue()

This function updates the available queue size in the router

class ReceiverDevice(simpyenv, id, gymenv, rec_arrivals=True, absolute_arrivals=False, rec_waits=True, debug=True, selector=None)

Bases: CyberWithChannelEnvSB_123_Experimentation.Interpreter

This is the ReceiverDevice class acting as the Data Aggregator/ DSO (Distribution System Operator)

getDone()

This function obtains the status of the episode if the agent has reached the goal or not

getInfo()

Returns a dict providing additional information on the environment’s state that may be useful for debugging but is not allowed to be used by a learning agent.

getObservation()

This function obtains the system state

getReward()

Will depend on the percentage of successfully packet received (WE WILL FURTHER incorprorate other resilience metric)

put(pkt)

This function process the receiver packets from all the Data Concentrator of each zone

reset()

This method is invoked when the environment is reset – override it with your initialization tasks if you feel like it.

class SenderDevice(env, id, adist, sdist, initial_delay=0, finish=inf, flow_id=0, debug=True, packet_size=None)

Bases: envs.simpy_env.SimComponentsDynamic.PacketGenerator

This is the SenderDevice class acting as the data concentrator that collects data from the smart meter within a zone and forward to the DSO

sendCommand()

This is the process to send command generated based on the switching control action created in the distribution feeder

senderProcess()

This is the process to send the monitoring packet

class WiredChannel(env, src, dest)

Bases: CyberWithChannelEnvSB_123_Experimentation.CyberEnv.Channel

run()
class WirelessChannel(env, src, dest)

Bases: CyberWithChannelEnvSB_123_Experimentation.CyberEnv.Channel

run()
reinitialize_complex_network(G)

This function re-initialize the larger network for the IEEE 123 distribution feeder every time the episode starts

reinitialize_network()

This function re-initialize the network every time the episode starts

render(mode='human', close=False)

Renders the environment to stdout.

reset()

Resets the state of the environment and returns an initial observation.

step(action, result={}, pc_queue=<queue.Queue object>, cp_queue=<queue.Queue object>)

This function executes the selected action on the environment. In this environment the action are selection of the router and the next hop router to forward. Unless

action: The selected router and its interface selected for forwarding result: for storing the results pc_queue : queue used to forward physical env information to communication env cp_queue : queue used to forward cyber env information to physical env

threat_model(targets)

This function incorporates the threat model

class CyberWithChannelEnvSB_123_Experimentation.Interpreter

Bases: abc.ABC

An Interpreter is an instance that observes the system’s behavior by sniffing the packets received by the receiver and infers observations and rewards. This class serves as an abstract base class for all Interpreter implementations. When implementing an interpreter, the following three methods have to be overridden:

The following methods provide default implementations that you might also want to override depending on your use case:

getDone() → bool

Returns whether an episode has ended. Note:

Reinforcement learning problems do not have to be split into episodes. In this case, you do not have to override the default implementation as it always returns False.

getFeedback() → Tuple[Any, float, bool, Dict]

You may want to call this at the end of a frequency band assignment to get feedback for your learning agent. The return values are ordered like they need to be returned by the step() method of a gym environment. Returns:

A 4-tuple with the results of getObservation(), getReward(), getDone(), and getInfo()

getInfo() → Dict

Returns a dict providing additional information on the environment’s state that may be useful for debugging but is not allowed to be used by a learning agent.

abstract getObservation() → Any

Returns an observation of the system’s state.

abstract getReward() → float

Returns a reward that depends on the last channel assignment.

reset()

This method is invoked when the environment is reset – override it with your initialization tasks if you feel like it.

class CyberWithChannelEnvSB_123_Experimentation.Message(type: enum.Enum, args: Optional[Dict[str, Any]] = None)

Bases: object

A class used for the exchange of arbitrary messages between components. A Message can be used to simulate both asynchronous and synchronous function calls. Attributes:

type(Enum): An enumeration object that defines the message type args(Dict[str, Any]): A dictionary containing the message’s arguments eProcessed(Event): A SimPy event that is triggered when

setProcessed() is called. This is useful for simulating synchronous function calls and also allows for return values (an example is provided in setProcessed()).

setProcessed(returnValue: Optional[Any] = None)

Makes the eProcessed event succeed. Args:

returnValue: If specified, will be used as the value of the

eProcessed event.

Examples:

If returnValue is specified, SimPy processes can use Signals for simulating synchronous function calls with return values like this:

signal = Signal(myType, {"key", value})
gate.output.send(signal)
value = yield signal.eProcessed
# value now contains the returnValue that setProcessed() was called with
class CyberWithChannelEnvSB_123_Experimentation.StackMessageTypes(value)

Bases: enum.Enum

An enumeration of control message types to be used for the exchange of Message objects between network stack layers.

ASSIGN = 2
RECEIVE = 0
SEND = 1
CyberWithChannelEnvSB_123_Experimentation.constArrival()

Constant arrival rate of the packet

CyberWithChannelEnvSB_123_Experimentation.constArrival2()
CyberWithChannelEnvSB_123_Experimentation.constSize()

Constant packet size

CyberWithChannelEnvSB_123_Experimentation.constSize2()