please make sure that developer can open Modal dialog in nintex form.
when i am trying to open a modal dialog in nintex form, neither SP.SOD.execute('sp.ui.dialog.js', 'SP.UI.ModalDialog.showModalDialog', options);
is going to work.
When I googled it, I found that it is because Nintex in O365 is a APP and form is displayed in iframe.
Thanks for your suggestion.
I will update this idea if there are any updates to share.