QELAR - Revision history

Tiansi: /* Overview */

2012-05-23T18:18:47Z

Overview

← Older revision		Revision as of 18:18, 23 May 2012
Line 1:		Line 1:
	= Overview =		= Overview =
-	QELAR is a Q-learning-based energy-~~efficent~~ and lifetime-aware routing protocol. It is designed to address various issues related to underwater acoustic sensor networks (UW-ASNs). By learning the environment and evaluating an action-value function (Q-value), which gives the expected reward of taking an action in a given state, the distributed learning agent is able to make a decision automatically.	+	QELAR is a Q-learning-based energy-efficient and lifetime-aware routing protocol. It is designed to address various issues related to underwater acoustic sensor networks (UW-ASNs). By learning the environment and evaluating an action-value function (Q-value), which gives the expected reward of taking an action in a given state, the distributed learning agent is able to make a decision automatically.

	We find that Q-learning is very suitable in UW-ASNs in the following ways:		We find that Q-learning is very suitable in UW-ASNs in the following ways:

Tiansi at 05:00, 23 March 2012

2012-03-23T05:00:29Z

← Older revision		Revision as of 05:00, 23 March 2012
Line 1:		Line 1:
-	~~=Description=~~	+	= Overview =
-	== Overview ==	+
	QELAR is a Q-learning-based energy-efficent and lifetime-aware routing protocol. It is designed to address various issues related to underwater acoustic sensor networks (UW-ASNs). By learning the environment and evaluating an action-value function (Q-value), which gives the expected reward of taking an action in a given state, the distributed learning agent is able to make a decision automatically.		QELAR is a Q-learning-based energy-efficent and lifetime-aware routing protocol. It is designed to address various issues related to underwater acoustic sensor networks (UW-ASNs). By learning the environment and evaluating an action-value function (Q-value), which gives the expected reward of taking an action in a given state, the distributed learning agent is able to make a decision automatically.

Line 13:		Line 12:
	* '''General Framework.''' Q-learning is a framework that can be easily extended. We can easily integrate other factors such as end-to-end delay and node density for extension and can balance all the factors according to our need by tuning the parameters in the reward function.		* '''General Framework.''' Q-learning is a framework that can be easily extended. We can easily integrate other factors such as end-to-end delay and node density for extension and can balance all the factors according to our need by tuning the parameters in the reward function.

-	== Design ==	+	= Design =
	[[Image: rl.jpg\|thumb\|300px]]		[[Image: rl.jpg\|thumb\|300px]]
-	=== Q-learning ===	+	== Q-learning ==
	Q-learning is one type of Reinforcement Learning algorithms, by which a system can learn to achieve a goal in control problems based on its experience. An agent in RL chooses actions according to the current state of a system and the reinforcement it receives from the environment. Most RL algorithms are based on estimating value functions, functions of states (or of state-action pairs), which evaluate how good it is for the agent to be in a given state (or how good it is to perform an action in a given state).		Q-learning is one type of Reinforcement Learning algorithms, by which a system can learn to achieve a goal in control problems based on its experience. An agent in RL chooses actions according to the current state of a system and the reinforcement it receives from the environment. Most RL algorithms are based on estimating value functions, functions of states (or of state-action pairs), which evaluate how good it is for the agent to be in a given state (or how good it is to perform an action in a given state).

Line 25:		Line 24:
	where <math>s'</math> is the next state, <math>\alpha</math> and <math>\gamma</math> is the learning rate and future discount, respectively.		where <math>s'</math> is the next state, <math>\alpha</math> and <math>\gamma</math> is the learning rate and future discount, respectively.

-	=== Reward Function ===	+	== Reward Function ==
	The reward function in Q-learning determines the behavior of the learning agent. In QELAR, we consider the following three rewards:		The reward function in Q-learning determines the behavior of the learning agent. In QELAR, we consider the following three rewards:
	*'''The penalty of forwarding a packet.''' Forwarding a packet consumes energy, occupies channel bandwidth, and contributes to the delay. Therefore forwarding a packet should always receive a negative reward, which is <math>-g</math>.		*'''The penalty of forwarding a packet.''' Forwarding a packet consumes energy, occupies channel bandwidth, and contributes to the delay. Therefore forwarding a packet should always receive a negative reward, which is <math>-g</math>.
Line 44:		Line 43:
	where <math>\alpha_1</math> and <math>\alpha_2</math> are the constant weights that can be adjusted.		where <math>\alpha_1</math> and <math>\alpha_2</math> are the constant weights that can be adjusted.

-	=== Packet Format ===	+	== Packet Format ==
	[[Image: pf.jpg\|thumb\|300px]]		[[Image: pf.jpg\|thumb\|300px]]

Line 62:		Line 61:
	The last field '''Next Forwarder''' indicates that the ID of the node eligible to be the next forwarder.		The last field '''Next Forwarder''' indicates that the ID of the node eligible to be the next forwarder.

-	== Simulation Tools ==	+	= Simulation Tools =
	The simulation tool is originally developed on [http://www.isi.edu/nsnam/ns/ NS2], and is currently being moved to [http://www.omnetpp.org OMNet++].		The simulation tool is originally developed on [http://www.isi.edu/nsnam/ns/ NS2], and is currently being moved to [http://www.omnetpp.org OMNet++].

-	=== Downloads ===	+	== Downloads ==
	The simulation tool of QELAR is built based on		The simulation tool of QELAR is built based on
	*[http://www.omnetpp.org OMNet++ 4.1]: [http://laurel.engr.uconn.edu/~tiansi/tools/omnetpp-4.1-src.tgz omnetpp-4.1-src.tgz]		*[http://www.omnetpp.org OMNet++ 4.1]: [http://laurel.engr.uconn.edu/~tiansi/tools/omnetpp-4.1-src.tgz omnetpp-4.1-src.tgz]
Line 72:		Line 71:
	The QELAR tool is currently being ported to the platform of OMNet++. It is coming soon.		The QELAR tool is currently being ported to the platform of OMNet++. It is coming soon.

-	=== Installation Guide ===	+	== Installation Guide ==
	Will be provided upon release.		Will be provided upon release.

-	=== User Guide ===	+	== User Guide ==
	Will be provided upon release.		Will be provided upon release.
-
-	~~=People=~~
-	*[[Tiansi Hu]]
-	*[[Yu Han]]
-
-	~~=Related Publications=~~
-	*{{#lst:Publications\|thc4}}
-	*{{#lst:Publications\|thc3}}
-	*{{#lst:Publications\|thc2}}
-	*T. Hu and Y. Fei, “QELAR: A machine-learning-based adaptive routing protocol for energy efficient and lifetime-extended underwater sensor networks,” IEEE Trans. on Mobile Computing, vol. 9, no. 6, June 2010.
-	*P. Xie, Z. Zhou, Z. Peng, H. Yan, T. Hu, J.-H. Cui, Z. Shi, Y. Fei, and S. Zhou, Aqua-Sim: An NS-2 Based Simulator for Underwater Sensor Networks, IEEE/MTS Oceans 2009, Biloxi, Mississippi, Oct 2009.
-	*T. Hu and Y. Fei, “QELAR - A Q-learning-based energy-efficient and lifetime-aware routing protocol for underwater sensor networks,” in IEEE Int. Performance Computing & Communications Conf., Dec. 2008.

Yunsi: /* People */

2012-03-20T14:22:50Z

People

← Older revision		Revision as of 14:22, 20 March 2012
Line 80:		Line 80:
	=People=		=People=
	*[[Tiansi Hu]]		*[[Tiansi Hu]]
		+	*[[Yu Han]]

	=Related Publications=		=Related Publications=

Yunsi: /* Related Publications */

2012-03-20T14:22:12Z

Related Publications

← Older revision		Revision as of 14:22, 20 March 2012
Line 82:		Line 82:

	=Related Publications=		=Related Publications=
		+	*{{#lst:Publications\|thc4}}
		+	*{{#lst:Publications\|thc3}}
		+	*{{#lst:Publications\|thc2}}
	*T. Hu and Y. Fei, “QELAR: A machine-learning-based adaptive routing protocol for energy efficient and lifetime-extended underwater sensor networks,” IEEE Trans. on Mobile Computing, vol. 9, no. 6, June 2010.		*T. Hu and Y. Fei, “QELAR: A machine-learning-based adaptive routing protocol for energy efficient and lifetime-extended underwater sensor networks,” IEEE Trans. on Mobile Computing, vol. 9, no. 6, June 2010.
	*P. Xie, Z. Zhou, Z. Peng, H. Yan, T. Hu, J.-H. Cui, Z. Shi, Y. Fei, and S. Zhou, Aqua-Sim: An NS-2 Based Simulator for Underwater Sensor Networks, IEEE/MTS Oceans 2009, Biloxi, Mississippi, Oct 2009.		*P. Xie, Z. Zhou, Z. Peng, H. Yan, T. Hu, J.-H. Cui, Z. Shi, Y. Fei, and S. Zhou, Aqua-Sim: An NS-2 Based Simulator for Underwater Sensor Networks, IEEE/MTS Oceans 2009, Biloxi, Mississippi, Oct 2009.
	*T. Hu and Y. Fei, “QELAR - A Q-learning-based energy-efficient and lifetime-aware routing protocol for underwater sensor networks,” in IEEE Int. Performance Computing & Communications Conf., Dec. 2008.		*T. Hu and Y. Fei, “QELAR - A Q-learning-based energy-efficient and lifetime-aware routing protocol for underwater sensor networks,” in IEEE Int. Performance Computing & Communications Conf., Dec. 2008.

Tiansi at 18:27, 12 December 2011

2011-12-12T18:27:33Z

← Older revision		Revision as of 18:27, 12 December 2011
Line 30:		Line 30:
	*'''Residual energy.''' Lower reward should be given if the residual energy of either the sender or the receiver is low. Therefore, forwarding to a packet to a node with low residual energy can be avoided. The reward related to residual energy of Node <math>n</math> is defined as		*'''Residual energy.''' Lower reward should be given if the residual energy of either the sender or the receiver is low. Therefore, forwarding to a packet to a node with low residual energy can be avoided. The reward related to residual energy of Node <math>n</math> is defined as
	<math>		<math>
-	c(n)=\frac{E_{res}(n)}{E_{init}(n)}	+	c(n)=\frac{E_{res}(n)}{E_{init}(n)}.
	</math>		</math>
	*'''Energy distribution.''' Energy distribution should also be considered to ensure that the energy is distributed evenly among a group of sensor nodes, which includes the node <math>n</math> that holds a packet and all its direct neighbors in the transmission range. It is defined as		*'''Energy distribution.''' Energy distribution should also be considered to ensure that the energy is distributed evenly among a group of sensor nodes, which includes the node <math>n</math> that holds a packet and all its direct neighbors in the transmission range. It is defined as
	<math>		<math>
-	d(n)=\frac{2}{\pi}\arctan(E_{res}(n)-\overline{E}(n)).	+	d(n)=\frac{2}{\pi}\arctan(E_{res}(n)-\overline{E}(n)),
	</math>		</math>
	where <math>\overline{E}(n)</math> is the average residual energy in Node <math>n</math>'s group.		where <math>\overline{E}(n)</math> is the average residual energy in Node <math>n</math>'s group.
Line 82:		Line 82:

	=Related Publications=		=Related Publications=
-	*T. Hu and Y. Fei, “An adaptive and energy-efficient routing protocol based on machine learning for underwater delay tolerant networks,” in ACM/IEEE Int. Symp. on Modeling, Analysis, and Simulation of Computer & Telecommunication Systems (MASCOTS), Aug. 2010.
	*T. Hu and Y. Fei, “QELAR: A machine-learning-based adaptive routing protocol for energy efficient and lifetime-extended underwater sensor networks,” IEEE Trans. on Mobile Computing, vol. 9, no. 6, June 2010.		*T. Hu and Y. Fei, “QELAR: A machine-learning-based adaptive routing protocol for energy efficient and lifetime-extended underwater sensor networks,” IEEE Trans. on Mobile Computing, vol. 9, no. 6, June 2010.
	*P. Xie, Z. Zhou, Z. Peng, H. Yan, T. Hu, J.-H. Cui, Z. Shi, Y. Fei, and S. Zhou, Aqua-Sim: An NS-2 Based Simulator for Underwater Sensor Networks, IEEE/MTS Oceans 2009, Biloxi, Mississippi, Oct 2009.		*P. Xie, Z. Zhou, Z. Peng, H. Yan, T. Hu, J.-H. Cui, Z. Shi, Y. Fei, and S. Zhou, Aqua-Sim: An NS-2 Based Simulator for Underwater Sensor Networks, IEEE/MTS Oceans 2009, Biloxi, Mississippi, Oct 2009.
	*T. Hu and Y. Fei, “QELAR - A Q-learning-based energy-efficient and lifetime-aware routing protocol for underwater sensor networks,” in IEEE Int. Performance Computing & Communications Conf., Dec. 2008.		*T. Hu and Y. Fei, “QELAR - A Q-learning-based energy-efficient and lifetime-aware routing protocol for underwater sensor networks,” in IEEE Int. Performance Computing & Communications Conf., Dec. 2008.

Bnjiang at 17:17, 19 August 2011

2011-08-19T17:17:56Z

← Older revision		Revision as of 17:17, 19 August 2011
Line 61:		Line 61:

	The last field '''Next Forwarder''' indicates that the ID of the node eligible to be the next forwarder.		The last field '''Next Forwarder''' indicates that the ID of the node eligible to be the next forwarder.
-
-	~~== Related Publications ==~~
-	*T. Hu and Y. Fei, “QELAR: A machine-learning-based adaptive routing protocol for energy efficient and lifetime-extended underwater sensor networks,” IEEE Trans. on Mobile Computing, vol. 9, no. 6, June 2010.
-	*T. Hu and Y. Fei, “QELAR - A Q-learning-based energy-efficient and lifetime-aware routing protocol for underwater sensor networks,” in IEEE Int. Performance Computing & Communications Conf., Dec. 2008.

	== Simulation Tools ==		== Simulation Tools ==

Bnjiang: /* Publications */

2011-08-19T17:16:55Z

Publications

← Older revision		Revision as of 17:16, 19 August 2011
Line 85:		Line 85:
	*[[Tiansi Hu]]		*[[Tiansi Hu]]

-	=Publications=	+	=Related Publications=
	*T. Hu and Y. Fei, “An adaptive and energy-efficient routing protocol based on machine learning for underwater delay tolerant networks,” in ACM/IEEE Int. Symp. on Modeling, Analysis, and Simulation of Computer & Telecommunication Systems (MASCOTS), Aug. 2010.		*T. Hu and Y. Fei, “An adaptive and energy-efficient routing protocol based on machine learning for underwater delay tolerant networks,” in ACM/IEEE Int. Symp. on Modeling, Analysis, and Simulation of Computer & Telecommunication Systems (MASCOTS), Aug. 2010.
	*T. Hu and Y. Fei, “QELAR: A machine-learning-based adaptive routing protocol for energy efficient and lifetime-extended underwater sensor networks,” IEEE Trans. on Mobile Computing, vol. 9, no. 6, June 2010.		*T. Hu and Y. Fei, “QELAR: A machine-learning-based adaptive routing protocol for energy efficient and lifetime-extended underwater sensor networks,” IEEE Trans. on Mobile Computing, vol. 9, no. 6, June 2010.
	*P. Xie, Z. Zhou, Z. Peng, H. Yan, T. Hu, J.-H. Cui, Z. Shi, Y. Fei, and S. Zhou, Aqua-Sim: An NS-2 Based Simulator for Underwater Sensor Networks, IEEE/MTS Oceans 2009, Biloxi, Mississippi, Oct 2009.		*P. Xie, Z. Zhou, Z. Peng, H. Yan, T. Hu, J.-H. Cui, Z. Shi, Y. Fei, and S. Zhou, Aqua-Sim: An NS-2 Based Simulator for Underwater Sensor Networks, IEEE/MTS Oceans 2009, Biloxi, Mississippi, Oct 2009.
	*T. Hu and Y. Fei, “QELAR - A Q-learning-based energy-efficient and lifetime-aware routing protocol for underwater sensor networks,” in IEEE Int. Performance Computing & Communications Conf., Dec. 2008.		*T. Hu and Y. Fei, “QELAR - A Q-learning-based energy-efficient and lifetime-aware routing protocol for underwater sensor networks,” in IEEE Int. Performance Computing & Communications Conf., Dec. 2008.

Bnjiang: /* Publications */

2011-08-19T16:25:49Z

Publications

← Older revision		Revision as of 16:25, 19 August 2011
Line 86:		Line 86:

	=Publications=		=Publications=
		+	*T. Hu and Y. Fei, “An adaptive and energy-efficient routing protocol based on machine learning for underwater delay tolerant networks,” in ACM/IEEE Int. Symp. on Modeling, Analysis, and Simulation of Computer & Telecommunication Systems (MASCOTS), Aug. 2010.
		+	*T. Hu and Y. Fei, “QELAR: A machine-learning-based adaptive routing protocol for energy efficient and lifetime-extended underwater sensor networks,” IEEE Trans. on Mobile Computing, vol. 9, no. 6, June 2010.
		+	*P. Xie, Z. Zhou, Z. Peng, H. Yan, T. Hu, J.-H. Cui, Z. Shi, Y. Fei, and S. Zhou, Aqua-Sim: An NS-2 Based Simulator for Underwater Sensor Networks, IEEE/MTS Oceans 2009, Biloxi, Mississippi, Oct 2009.
		+	*T. Hu and Y. Fei, “QELAR - A Q-learning-based energy-efficient and lifetime-aware routing protocol for underwater sensor networks,” in IEEE Int. Performance Computing & Communications Conf., Dec. 2008.

Bnjiang: /* People */

2011-08-19T16:24:52Z

People

← Older revision		Revision as of 16:24, 19 August 2011
Line 83:		Line 83:

	=People=		=People=
		+	*[[Tiansi Hu]]
		+
	=Publications=		=Publications=

Bnjiang at 16:23, 19 August 2011

2011-08-19T16:23:25Z

← Older revision		Revision as of 16:23, 19 August 2011
Line 1:		Line 1:
		+	=Description=
	== Overview ==		== Overview ==
	QELAR is a Q-learning-based energy-efficent and lifetime-aware routing protocol. It is designed to address various issues related to underwater acoustic sensor networks (UW-ASNs). By learning the environment and evaluating an action-value function (Q-value), which gives the expected reward of taking an action in a given state, the distributed learning agent is able to make a decision automatically.		QELAR is a Q-learning-based energy-efficent and lifetime-aware routing protocol. It is designed to address various issues related to underwater acoustic sensor networks (UW-ASNs). By learning the environment and evaluating an action-value function (Q-value), which gives the expected reward of taking an action in a given state, the distributed learning agent is able to make a decision automatically.
Line 80:		Line 81:
	=== User Guide ===		=== User Guide ===
	Will be provided upon release.		Will be provided upon release.
		+
		+	=People=
		+	=Publications=