Abstract

Machine-learning-based solutions are showing promising results for several critical issues in large-scale optical networks. Alarm (caused by failure, disaster, etc.) prediction is an important use-case, where machine learning can assist in predicting events, ahead of time. Accurate prediction enables network administrators to undertake preventive measures. For such alarm prediction applications, high-quality data sets for training and testing are crucial. However, the collected performance and alarm data from large-scale optical networks are often dirty, i.e., these data are incomplete, inconsistent, and lack certain behaviors or trends. Such data are likely to contain several errors, when collected from old-fashioned optical equipment, in particular. Even after appropriate data preprocessing, feature distribution can be extremely unbalanced, limiting the performance of machine learning algorithms. This paper demonstrates a Dirty-data-based Alarm Prediction (DAP) method for Self-Optimizing Optical Networks (SOONs). Experimental results on a commercial large-scale field topology with 274 nodes and 487 links demonstrate that the proposed DAP method can achieve high accuracy for different types of alarms.

© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

Full Article  |  PDF Article
OSA Recommended Articles
SOON: self-optimizing optical networks with machine learning

Yongli Zhao, Boyuan Yan, Dongmei Liu, Yongqi He, Dajiang Wang, and Jie Zhang
Opt. Express 26(22) 28713-28726 (2018)

Experimental Demonstration of a Lightpath Management Procedure Based on a Hierarchical Architecture

P. Giardina, N. Sambo, M. Dallaglio, G. Bernini, G. Carrozzo, F. Cugini, and P. Castoldi
J. Opt. Commun. Netw. 10(4) 386-395 (2018)

Machine Learning for Network Automation: Overview, Architecture, and Applications [Invited Tutorial]

Danish Rafique and Luis Velasco
J. Opt. Commun. Netw. 10(10) D126-D143 (2018)

References

  • View by:
  • |
  • |
  • |

  1. Y. R. Zhou, K. Smith, M. Gilson, J. Chen, W. Pan, Y. Chang, S. Wu, S. Wu, and I. Davis, “Demonstration of real-time 400G single-carrier ultra-efficient 1.2 Tb/s superchannel over large Aeff ultra-low loss terrestrial fiber of 150 km single span and 250 km (2×125 km spans) using only EDFA amplification,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2018), paper M1E.4.
  2. M. Mazur, A. Lorences-Riesgo, J. Schroder, P. A. Andrekson, and M. Karlsson, “10 Tb/s PM-64QAM self-homodyne comb-based superchannel transmission with 4% shared pilot tone overhead,” J. Lightwave Technol. 36(16), 3176–3184 (2018).
    [Crossref]
  3. S. Rahman, T. Ahmed, M. Huynh, M. Tornatore, and B. Mukherjee, “Auto-scaling VNFs using machine learning to improve qos and reduce cost,” in Proc., IEEE Intl. Conf. on Commun., Kansas City, USA, 2018, pp. 1–6.
    [Crossref]
  4. B. Yan, Y. Zhao, Y. Li, X. Yu, J. Zhang, Y. Wang, L. Yan, and S. Rahman, “Actor-critic-based resource allocation for multi-modal optical networks,” in Proc. IEEE GLOBCOM Workshop on Machine Learning for Communications, Abu Dhabi, UAE, Dec. 2018.
    [Crossref]
  5. Z. Wang, M. Zhang, D. Wang, C. Song, M. Liu, J. Li, L. Lou, and Z. Liu, “Failure prediction using machine learning and time series in optical network,” Opt. Express 25(16), 18553–18565 (2017).
    [Crossref] [PubMed]
  6. L. Barletta, A. Giusti, C. A. Rottondi, and M. Tornatore, “QoT estimation for unestablished lighpaths using machine learning,” in OFC, Th1J.1, 2017.
  7. T. Panayiotou, S. P. Chatzis, and G. Ellinas, “Leveraging statistical machine learning to address failure localization in optical networks,” J. Opt. Commun. Netw. 10(3), 162–173 (2018).
    [Crossref]
  8. W. Kim, B. Choi, E. Hong, S. Kim, and D. Lee, “A taxonomy of dirty data,” Data Min. Knowl. Discov. 7(1), 81–99 (2003).
    [Crossref]
  9. Q. Wang, Z. Luo, J. Huang, Y. Feng, and Z. Liu, “A Novel Ensemble Method for Imbalanced Data Learning: Bagging of Extrapolation-SMOTE SVM,” Comput. Intell. Neurosci. 2017, 1827016 (2017).
    [Crossref] [PubMed]
  10. H. He and E. A. Garcia, “Learning from Imbalanced Data,” IEEE T Knowl. Data En. 21(9), 1263–1284 (2009).
  11. N. V. Chawla, N. Japkowicz, and A. Kotcz, “Editorial: special issue on learning from imbalanced data sets,” SIGKDD Explor. 6(1), 1–6 (2004).
    [Crossref]
  12. N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: synthetic minority over-sampling technique,” J. Artif. Intell. Res. 16, 321–357 (2002).
    [Crossref]
  13. H. Han, W. Wang, and B. Mao, “Borderline-SMOTE: A new over-sampling method in imbalanced data sets learning,” in International Conference on Intelligent Computing (Springer 2005), pp. 878–887.
    [Crossref]
  14. Y. Mi, “Imbalanced classification based on active learning SMOTE,” J Appl. Sci. En. Technol. 5(3), 944–949 (2013).
  15. G. Douzas and F. Bação, “Geometric SMOTE: Effective oversampling for imbalanced learning through a geometric extension of SMOTE,” arXiv 1709.07377 (2017).
  16. B. Yan, Y. Zhao, W. Wang, L. Yan, Y. Wang, J. Liu, S. Zhang, D. Liu, Y. Lin, H. Zheng, and J. Zhang, “First demonstration of machine-learning-based self-optimizing optical networks (SOON) running on commercial equipment, ” in European Conference on Optical Communication, Roma, Italy, 23–27 Sept. TuDS.3, 2018.
    [Crossref]
  17. Y. Zhao, B. Yan, D. Liu, Y. He, D. Wang, and J. Zhang, “SOON: self-optimizing optical networks with machine learning,” Opt. Express 26(22), 28713–28726 (2018).
    [Crossref] [PubMed]
  18. B. Yan, Y. Zhao, Y. Li, X. Yu, J. Zhang, and H. Z. Yilin, “First Demonstration of Imbalanced Data Learning-Based Failure Prediction in Self-Optimizing Optical Networks with Large Scale Field Topology,” in 2018 Asia Communications and Photonics Conference (ACP), Hangzhou, China, pp. 1–4, 2018.
  19. X. Zheng and N. Hua, “achieving inter-connection in multi-domain heterogeneous optical network: from PCE to SDON,” in Asia Communications and Photonics Conference 2013, OSA Technical Digest (online) (Optical Society of America, 2013), paper AW4I.2.
  20. Open Networking Foundation, “Open Networking Operation System,” https://onosproject.org/ .
  21. Google Inc, “TensorFlow,” https://www.tensorflow.org/ .
  22. The PostgreSQL Global Development Group, “PostgreSQL: The world’s most advanced open source relational database,” https://www.postgresql.org/ .
  23. J. Han, M. Kamber, and J. Pei, Data Mining: Concepts and Techniques (Elsevier, 2011), Chap. 3.
  24. R. M. Morais and J. Pedro, “Machine Learning Models for Estimating Quality of Transmission in DWDM Networks,” J. Opt. Commun. Netw. 10(10), D84–D99 (2018).
    [Crossref]
  25. H. Robbins and S. Monroe, “A stochastic approximation method,” Ann. Math. Stat. 22(3), 400–407 (1951).
    [Crossref]

2018 (4)

2017 (2)

Z. Wang, M. Zhang, D. Wang, C. Song, M. Liu, J. Li, L. Lou, and Z. Liu, “Failure prediction using machine learning and time series in optical network,” Opt. Express 25(16), 18553–18565 (2017).
[Crossref] [PubMed]

Q. Wang, Z. Luo, J. Huang, Y. Feng, and Z. Liu, “A Novel Ensemble Method for Imbalanced Data Learning: Bagging of Extrapolation-SMOTE SVM,” Comput. Intell. Neurosci. 2017, 1827016 (2017).
[Crossref] [PubMed]

2013 (1)

Y. Mi, “Imbalanced classification based on active learning SMOTE,” J Appl. Sci. En. Technol. 5(3), 944–949 (2013).

2009 (1)

H. He and E. A. Garcia, “Learning from Imbalanced Data,” IEEE T Knowl. Data En. 21(9), 1263–1284 (2009).

2004 (1)

N. V. Chawla, N. Japkowicz, and A. Kotcz, “Editorial: special issue on learning from imbalanced data sets,” SIGKDD Explor. 6(1), 1–6 (2004).
[Crossref]

2003 (1)

W. Kim, B. Choi, E. Hong, S. Kim, and D. Lee, “A taxonomy of dirty data,” Data Min. Knowl. Discov. 7(1), 81–99 (2003).
[Crossref]

2002 (1)

N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: synthetic minority over-sampling technique,” J. Artif. Intell. Res. 16, 321–357 (2002).
[Crossref]

1951 (1)

H. Robbins and S. Monroe, “A stochastic approximation method,” Ann. Math. Stat. 22(3), 400–407 (1951).
[Crossref]

Ahmed, T.

S. Rahman, T. Ahmed, M. Huynh, M. Tornatore, and B. Mukherjee, “Auto-scaling VNFs using machine learning to improve qos and reduce cost,” in Proc., IEEE Intl. Conf. on Commun., Kansas City, USA, 2018, pp. 1–6.
[Crossref]

Andrekson, P. A.

Bowyer, K. W.

N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: synthetic minority over-sampling technique,” J. Artif. Intell. Res. 16, 321–357 (2002).
[Crossref]

Chatzis, S. P.

Chawla, N. V.

N. V. Chawla, N. Japkowicz, and A. Kotcz, “Editorial: special issue on learning from imbalanced data sets,” SIGKDD Explor. 6(1), 1–6 (2004).
[Crossref]

N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: synthetic minority over-sampling technique,” J. Artif. Intell. Res. 16, 321–357 (2002).
[Crossref]

Choi, B.

W. Kim, B. Choi, E. Hong, S. Kim, and D. Lee, “A taxonomy of dirty data,” Data Min. Knowl. Discov. 7(1), 81–99 (2003).
[Crossref]

Ellinas, G.

Feng, Y.

Q. Wang, Z. Luo, J. Huang, Y. Feng, and Z. Liu, “A Novel Ensemble Method for Imbalanced Data Learning: Bagging of Extrapolation-SMOTE SVM,” Comput. Intell. Neurosci. 2017, 1827016 (2017).
[Crossref] [PubMed]

Garcia, E. A.

H. He and E. A. Garcia, “Learning from Imbalanced Data,” IEEE T Knowl. Data En. 21(9), 1263–1284 (2009).

Hall, L. O.

N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: synthetic minority over-sampling technique,” J. Artif. Intell. Res. 16, 321–357 (2002).
[Crossref]

Han, H.

H. Han, W. Wang, and B. Mao, “Borderline-SMOTE: A new over-sampling method in imbalanced data sets learning,” in International Conference on Intelligent Computing (Springer 2005), pp. 878–887.
[Crossref]

He, H.

H. He and E. A. Garcia, “Learning from Imbalanced Data,” IEEE T Knowl. Data En. 21(9), 1263–1284 (2009).

He, Y.

Hong, E.

W. Kim, B. Choi, E. Hong, S. Kim, and D. Lee, “A taxonomy of dirty data,” Data Min. Knowl. Discov. 7(1), 81–99 (2003).
[Crossref]

Huang, J.

Q. Wang, Z. Luo, J. Huang, Y. Feng, and Z. Liu, “A Novel Ensemble Method for Imbalanced Data Learning: Bagging of Extrapolation-SMOTE SVM,” Comput. Intell. Neurosci. 2017, 1827016 (2017).
[Crossref] [PubMed]

Huynh, M.

S. Rahman, T. Ahmed, M. Huynh, M. Tornatore, and B. Mukherjee, “Auto-scaling VNFs using machine learning to improve qos and reduce cost,” in Proc., IEEE Intl. Conf. on Commun., Kansas City, USA, 2018, pp. 1–6.
[Crossref]

Japkowicz, N.

N. V. Chawla, N. Japkowicz, and A. Kotcz, “Editorial: special issue on learning from imbalanced data sets,” SIGKDD Explor. 6(1), 1–6 (2004).
[Crossref]

Karlsson, M.

Kegelmeyer, W. P.

N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: synthetic minority over-sampling technique,” J. Artif. Intell. Res. 16, 321–357 (2002).
[Crossref]

Kim, S.

W. Kim, B. Choi, E. Hong, S. Kim, and D. Lee, “A taxonomy of dirty data,” Data Min. Knowl. Discov. 7(1), 81–99 (2003).
[Crossref]

Kim, W.

W. Kim, B. Choi, E. Hong, S. Kim, and D. Lee, “A taxonomy of dirty data,” Data Min. Knowl. Discov. 7(1), 81–99 (2003).
[Crossref]

Kotcz, A.

N. V. Chawla, N. Japkowicz, and A. Kotcz, “Editorial: special issue on learning from imbalanced data sets,” SIGKDD Explor. 6(1), 1–6 (2004).
[Crossref]

Lee, D.

W. Kim, B. Choi, E. Hong, S. Kim, and D. Lee, “A taxonomy of dirty data,” Data Min. Knowl. Discov. 7(1), 81–99 (2003).
[Crossref]

Li, J.

Liu, D.

Liu, M.

Liu, Z.

Z. Wang, M. Zhang, D. Wang, C. Song, M. Liu, J. Li, L. Lou, and Z. Liu, “Failure prediction using machine learning and time series in optical network,” Opt. Express 25(16), 18553–18565 (2017).
[Crossref] [PubMed]

Q. Wang, Z. Luo, J. Huang, Y. Feng, and Z. Liu, “A Novel Ensemble Method for Imbalanced Data Learning: Bagging of Extrapolation-SMOTE SVM,” Comput. Intell. Neurosci. 2017, 1827016 (2017).
[Crossref] [PubMed]

Lorences-Riesgo, A.

Lou, L.

Luo, Z.

Q. Wang, Z. Luo, J. Huang, Y. Feng, and Z. Liu, “A Novel Ensemble Method for Imbalanced Data Learning: Bagging of Extrapolation-SMOTE SVM,” Comput. Intell. Neurosci. 2017, 1827016 (2017).
[Crossref] [PubMed]

Mao, B.

H. Han, W. Wang, and B. Mao, “Borderline-SMOTE: A new over-sampling method in imbalanced data sets learning,” in International Conference on Intelligent Computing (Springer 2005), pp. 878–887.
[Crossref]

Mazur, M.

Mi, Y.

Y. Mi, “Imbalanced classification based on active learning SMOTE,” J Appl. Sci. En. Technol. 5(3), 944–949 (2013).

Monroe, S.

H. Robbins and S. Monroe, “A stochastic approximation method,” Ann. Math. Stat. 22(3), 400–407 (1951).
[Crossref]

Morais, R. M.

Mukherjee, B.

S. Rahman, T. Ahmed, M. Huynh, M. Tornatore, and B. Mukherjee, “Auto-scaling VNFs using machine learning to improve qos and reduce cost,” in Proc., IEEE Intl. Conf. on Commun., Kansas City, USA, 2018, pp. 1–6.
[Crossref]

Panayiotou, T.

Pedro, J.

Rahman, S.

S. Rahman, T. Ahmed, M. Huynh, M. Tornatore, and B. Mukherjee, “Auto-scaling VNFs using machine learning to improve qos and reduce cost,” in Proc., IEEE Intl. Conf. on Commun., Kansas City, USA, 2018, pp. 1–6.
[Crossref]

Robbins, H.

H. Robbins and S. Monroe, “A stochastic approximation method,” Ann. Math. Stat. 22(3), 400–407 (1951).
[Crossref]

Schroder, J.

Song, C.

Tornatore, M.

S. Rahman, T. Ahmed, M. Huynh, M. Tornatore, and B. Mukherjee, “Auto-scaling VNFs using machine learning to improve qos and reduce cost,” in Proc., IEEE Intl. Conf. on Commun., Kansas City, USA, 2018, pp. 1–6.
[Crossref]

Wang, D.

Wang, Q.

Q. Wang, Z. Luo, J. Huang, Y. Feng, and Z. Liu, “A Novel Ensemble Method for Imbalanced Data Learning: Bagging of Extrapolation-SMOTE SVM,” Comput. Intell. Neurosci. 2017, 1827016 (2017).
[Crossref] [PubMed]

Wang, W.

H. Han, W. Wang, and B. Mao, “Borderline-SMOTE: A new over-sampling method in imbalanced data sets learning,” in International Conference on Intelligent Computing (Springer 2005), pp. 878–887.
[Crossref]

Wang, Z.

Yan, B.

Zhang, J.

Zhang, M.

Zhao, Y.

Ann. Math. Stat. (1)

H. Robbins and S. Monroe, “A stochastic approximation method,” Ann. Math. Stat. 22(3), 400–407 (1951).
[Crossref]

Comput. Intell. Neurosci. (1)

Q. Wang, Z. Luo, J. Huang, Y. Feng, and Z. Liu, “A Novel Ensemble Method for Imbalanced Data Learning: Bagging of Extrapolation-SMOTE SVM,” Comput. Intell. Neurosci. 2017, 1827016 (2017).
[Crossref] [PubMed]

Data Min. Knowl. Discov. (1)

W. Kim, B. Choi, E. Hong, S. Kim, and D. Lee, “A taxonomy of dirty data,” Data Min. Knowl. Discov. 7(1), 81–99 (2003).
[Crossref]

IEEE T Knowl. Data En. (1)

H. He and E. A. Garcia, “Learning from Imbalanced Data,” IEEE T Knowl. Data En. 21(9), 1263–1284 (2009).

J Appl. Sci. En. Technol. (1)

Y. Mi, “Imbalanced classification based on active learning SMOTE,” J Appl. Sci. En. Technol. 5(3), 944–949 (2013).

J. Artif. Intell. Res. (1)

N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: synthetic minority over-sampling technique,” J. Artif. Intell. Res. 16, 321–357 (2002).
[Crossref]

J. Lightwave Technol. (1)

J. Opt. Commun. Netw. (2)

Opt. Express (2)

SIGKDD Explor. (1)

N. V. Chawla, N. Japkowicz, and A. Kotcz, “Editorial: special issue on learning from imbalanced data sets,” SIGKDD Explor. 6(1), 1–6 (2004).
[Crossref]

Other (13)

Y. R. Zhou, K. Smith, M. Gilson, J. Chen, W. Pan, Y. Chang, S. Wu, S. Wu, and I. Davis, “Demonstration of real-time 400G single-carrier ultra-efficient 1.2 Tb/s superchannel over large Aeff ultra-low loss terrestrial fiber of 150 km single span and 250 km (2×125 km spans) using only EDFA amplification,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2018), paper M1E.4.

L. Barletta, A. Giusti, C. A. Rottondi, and M. Tornatore, “QoT estimation for unestablished lighpaths using machine learning,” in OFC, Th1J.1, 2017.

S. Rahman, T. Ahmed, M. Huynh, M. Tornatore, and B. Mukherjee, “Auto-scaling VNFs using machine learning to improve qos and reduce cost,” in Proc., IEEE Intl. Conf. on Commun., Kansas City, USA, 2018, pp. 1–6.
[Crossref]

B. Yan, Y. Zhao, Y. Li, X. Yu, J. Zhang, Y. Wang, L. Yan, and S. Rahman, “Actor-critic-based resource allocation for multi-modal optical networks,” in Proc. IEEE GLOBCOM Workshop on Machine Learning for Communications, Abu Dhabi, UAE, Dec. 2018.
[Crossref]

B. Yan, Y. Zhao, Y. Li, X. Yu, J. Zhang, and H. Z. Yilin, “First Demonstration of Imbalanced Data Learning-Based Failure Prediction in Self-Optimizing Optical Networks with Large Scale Field Topology,” in 2018 Asia Communications and Photonics Conference (ACP), Hangzhou, China, pp. 1–4, 2018.

X. Zheng and N. Hua, “achieving inter-connection in multi-domain heterogeneous optical network: from PCE to SDON,” in Asia Communications and Photonics Conference 2013, OSA Technical Digest (online) (Optical Society of America, 2013), paper AW4I.2.

Open Networking Foundation, “Open Networking Operation System,” https://onosproject.org/ .

Google Inc, “TensorFlow,” https://www.tensorflow.org/ .

The PostgreSQL Global Development Group, “PostgreSQL: The world’s most advanced open source relational database,” https://www.postgresql.org/ .

J. Han, M. Kamber, and J. Pei, Data Mining: Concepts and Techniques (Elsevier, 2011), Chap. 3.

H. Han, W. Wang, and B. Mao, “Borderline-SMOTE: A new over-sampling method in imbalanced data sets learning,” in International Conference on Intelligent Computing (Springer 2005), pp. 878–887.
[Crossref]

G. Douzas and F. Bação, “Geometric SMOTE: Effective oversampling for imbalanced learning through a geometric extension of SMOTE,” arXiv 1709.07377 (2017).

B. Yan, Y. Zhao, W. Wang, L. Yan, Y. Wang, J. Liu, S. Zhang, D. Liu, Y. Lin, H. Zheng, and J. Zhang, “First demonstration of machine-learning-based self-optimizing optical networks (SOON) running on commercial equipment, ” in European Conference on Optical Communication, Roma, Italy, 23–27 Sept. TuDS.3, 2018.
[Crossref]

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (10)

Fig. 1
Fig. 1 (a) SOON architecture and (b) SOON deployment scenario.
Fig. 2
Fig. 2 SOON 2.0: (a) ML model state machine, and (b) model training and application process.
Fig. 3
Fig. 3 DAP procedure.
Fig. 4
Fig. 4 Demonstration of collected performance data.
Fig. 5
Fig. 5 Binary classification cases, (a) SMOTE data augmentation, (b) borderline-SMOTE data augmentation, and (c) scenarios where traditional data augmentation methods do not work.
Fig. 6
Fig. 6 Data augmentation of limited Gaussian noise.
Fig. 7
Fig. 7 Experimental scenario: (a) Telecommunications room, (b) GPU server, (c) computing server and storage server, (d) large-scale field network with 274 nodes and 487 links, and (e) history-alarm distribution.
Fig. 8
Fig. 8 Experimental testbed: (a) State machine captured by Wireshark and (b) TensorBoard webpage.
Fig. 9
Fig. 9 Experimental results: (a) loss variation with iteration and (b) prediction precision.
Fig. 10
Fig. 10 Prediction error analysis.

Equations (4)

Equations on this page are rendered with MathJax. Learn more.

p k = 6 2π (min(I)a) exp( 9 (2x2 i k +min(I)a) 2 2 (min(I)a) 2 )
X=( x 1 Type value , x 2 , x 3 ,..., x 12 , x 13 Performance value )
x 1 ={ 0, IN_PWR_LOW 0.5, OUT_PWR_ABN 1, R_LOS
y{ [0,th), No failure [th,1], Failure

Metrics