Merge branch 'v2' of https://github.com/proddy/EMS-ESP into v2

README.md

@@ -21,19 +21,39 @@ Note: Version 2.0 is not backward compatible with v1.0. The File system structur
 
 - A web interface built using React and TypeScript to be secure and cross-browser compatible. Each restful endpoint is protected and issues a JWT which is then sent using Bearer Authentication. Implements a Web captive portal. On first installs EMS-ESP starts an Access Point where system settings can be configured. Note, this is still in a separate repo and pending a merge into this project.
   
-- A new console. Like 1.9.x it works with both Serial and Telnet but a lot more intuitive behaving like a Linux shell and secure. Multiple telnet sessions are supported now but watch out for slow connections and low memory. A password is need to change any settings. You can use TAB to auto-complete commands, ctrl-L, ctrl-U and the other typical console type shortcuts. Some key commands:
+- A new console. Like in version 1.9 it works with both Serial and Telnet but a lot more intuitive, behaving similar to a Linux-style shell. It supports multiple connections and any commands that effect the behavior of EMS-ESP are secure behind an admin password. You can use TAB to auto-complete commands, ctrl-L, ctrl-U and the other typical console type shortcuts. ctrl-D to exit the current menu. Some other important commands:
     * `help` lists the commands and keywords
-    * some commands take you into a new context, a bit like a sub-menu. e.g. `system`, `mqtt`, `thermostat`. Use `help` to show which commands this context has and `exit` to get back to the root.
+    * some commands take you into a new context, a bit like a sub-menu. e.g. `system`, `mqtt`, `thermostat`. Use `help` to show which commands this context has and `exit` or CTRL-D to get back to the root.
     * To change a setting use the `set` command. Typing `set` shows the current settings.
     * `show` shows the data specific to the context you're in.
     * `su` to switch to Admin which enables more commands such as most of the `set` commands. The default password is "neo" which can be changed with `passwd` from the system menu. When in Admin mode the command prompt switches from `$` to `#`.
-    * `log` sets the logging. `log off` disables logging. Use `log trace` to see the telegram traffic and `log debug` for very verbose logging. To watch a specific telegram ID or device ID use `log trace [id]`.
+    * `log` sets the logging. `log off` disables logging. Use `log trace` to see the telegram traffic and `log debug` for very verbose logging. To watch a specific telegram ID or device ID use `log trace [full|raw| [id]`.
 
-- There is no "serial mode" anymore like with version 1.9. When the Wifi cannot connect to the SSID it will automatically enter a "safe" mode where the Serial console is activated, baud 115200. Note Serial is always available on the ESP32 because it has 2 UARTs.
+- There is no "serial mode" anymore like with version 1.9. When the Wifi cannot connect to the SSID it will automatically enter a "safe" mode where the Serial console is activated, baud 115200. Note Serial is always available on the ESP32 because it has multiple UARTs.
 
 - LED behaves like in 1.9. A solid LED means good connection and EMS data is coming in. A slow pulse means either the WiFi or the EMS bus is not connected. A very fast pulse is when the system is booting up and configuring itself.
 
-- on a new install you will want to enter `su` and then go to the `system` context. Use `set wifi ...` to set the network up. Then go to the `mqtt` context to set the mqtt up.
+### Setting up for the first time:
+
+ - connect the ESP8266/ESP32 via USB and enter via the serial/com port with baud 115200
+ - type `su` and use the default password *neo*
+ - type `system` to go to the system sub-menu (which we call a context)
+ - type `set` to see the current settings
+ - use `set wifi` to change the SSID and password. Remember TAB auto-completes the command
+ - CTRL-D to get back to the root
+ - use `mqtt` to enter the MQTT menu. Same approach using the `set` command to set the MQTT IP and any credentials. CTRL-D to get back to the root.
+ - use `ems` to change to the EMS BUS tx mode if you find Tx is not working.
+ - reboot and next time use the Telnet via WiFi to connect as the serial mode will be disabled.
+
+### Debugging
+ - Turn on logging with either `log all` or `log trace` or `log debug`
+ - Error messages are shown in the color red
+ - type `show` from the main root to see if any data has come in
+ - if not, go to the `ems` context and type `show` which will display some EMS bus stats
+ - use the `refresh` command to fetch new data from the EMS bus
+ - use `scan devices` or `scan devices deep` to locate devices on the EMS bus. If any are unknown please report back to the project so we can update our EMS device library.
+
+
 
 # Full Console Commands
 
@@ -44,7 +64,7 @@ common commands available in all contexts:
   log [level] [full|raw] [trace ID]
   su
 
-(top level)
+(top root level)
 	refresh
 	show
 	show version
@@ -124,25 +144,19 @@ thermostat
 ### **Known issues, bugs and improvements currently working on**
 
 ```
-TODO ESP32 - when saving SPIFFS the UART stop and restart() functions need to flush queue to avoid miss fires
-TODO sometimes with tx_mode 0 there are a few CRC errors due to collision when waiting for a BRK signal.
+TODO Optimized ESP8266 and ESP32 UART code (via Michael)
 TODO console auto-complete with 'set' command in the system context is not showing all commands, only the hostname.
 ```
 
-### **Features to add next**
-
-```
-TODO validate 0xE9 with data from Koen. (https://github.com/proddy/EMS-ESP/issues/382)
-```
-
 ### **To tidy up in code later**
 
 ```
 TODO replace vectors of class objects with shared pointers and use emplace_back since it instantiates during construction. It may have a performance gain.
 TODO make more use of comparison operators in the Telegram class e.g. the compare like "friend inline bool operator==(const Telegram & lhs, const Telegram & rhs)"
+TODO replace read_value with C++ Templates per data type
 TODO exit from serial should be prevented? Because you never can really exit, just close it.
 TODO add real unit tests using platformio's test bed (https://docs.platformio.org/en/latest/plus/pio-remote.html)
-TODO See if it's easier to use timers instead of millis() timers, using https://github.com/esp8266/Arduino/blob/master/libraries/esp8266/examples/BlinkPolledTimeout/BlinkPolledTimeout.ino
+TODO See if it's easier to use timers instead of millis() based timers, using https://github.com/esp8266/Arduino/blob/master/libraries/esp8266/examples/BlinkPolledTimeout/BlinkPolledTimeout.ino
 ```
 
 ### **These features to add next**
@@ -151,5 +165,6 @@ TODO See if it's easier to use timers instead of millis() timers, using https://
 TODO merge in the web code which has the Captive AP and better wifi reconnect logic. Use IPV6 and NTP from lwip2
 TODO decide if I want to port over the shower one-shot cold water logic. Don't think its used.
 TODO when doing show in telnet, should we sort the ems devices?
+TODO validate 0xE9 with data from Koen. (https://github.com/proddy/EMS-ESP/issues/382)
 
 ```

src/emsesp.cpp

@@ -110,22 +110,22 @@ void EMSESP::trace_watch_id(uint16_t trace_watch_id) {
     }
 }
 
-// show the Rx and Tx queues
+// show the EMS bus status plus both Rx and Tx queues
 void EMSESP::show_emsbus(uuid::console::Shell & shell) {
-    // EMS bus specific
+    // EMS bus information
     if (rxservice_.bus_connected()) {
         uint8_t success_rate = 0;
         if (rxservice_.telegram_error_count()) {
             success_rate = ((float)rxservice_.telegram_error_count() / (float)rxservice_.telegram_count()) * 100;
         }
 
-        shell.printfln(F("EMS Bus protocol: %s, #telegrams received: %d, #Read requests sent: %d, #Write requests sent: %d, #CRC errors: %d (%d%%)"),
-                       EMSbus::is_ht3() ? F("HT3") : F("Buderus"),
-                       rxservice_.telegram_count(),
-                       txservice_.telegram_read_count(),
-                       txservice_.telegram_write_count(),
-                       rxservice_.telegram_error_count(),
-                       success_rate);
+        shell.printfln(F("EMS Bus info:"));
+        shell.printfln(F("  Bus protocol: %s"), EMSbus::is_ht3() ? F("HT3") : F("Buderus"));
+        shell.printfln(F("  #telegrams received: %d"), rxservice_.telegram_count());
+        shell.printfln(F("  #read requests sent: %d"), txservice_.telegram_read_count());
+        shell.printfln(F("  #write requests sent: %d"), txservice_.telegram_write_count());
+        shell.printfln(F("  #CRC errors: %d (%d%%)"), rxservice_.telegram_error_count(), success_rate);
+        shell.printfln(F("  #Tx fails: %d"), txservice_.telegram_fail_count());
     } else {
         shell.printfln(F("EMS Bus is disconnected"));
     }
@@ -570,21 +570,21 @@ void EMSESP::incoming_telegram(uint8_t * data, const uint8_t length) {
     }
 
     // are we waiting for a response from a recent Tx Read or Write?
+    bool tx_successful = false;
     if (EMSbus::tx_waiting()) {
-        // if it's a single byte 1 or 4 then its maybe a response from the last write action
         EMSbus::tx_waiting(false); // reset Tx wait state
+
+        // if it's a single byte 1 or 4 then its maybe a response from the last write action
         if (length == 1) {
             if (first_value == TxService::TX_WRITE_SUCCESS) {
                 LOG_DEBUG(F("Last Tx write successful. Sending read request."));
                 txservice_.increment_telegram_write_count(); // last tx/write was confirmed ok
                 txservice_.send_poll();                      // close the bus
-                txservice_.post_send_query();                // send type_id to last destination
+                txservice_.post_send_query();                // follow up with any post-read
+                tx_successful = true;
             } else if (first_value == TxService::TX_WRITE_FAIL) {
                 LOG_ERROR(F("Last Tx write rejected by host"));
                 txservice_.send_poll(); // close the bus
-            } else {
-                // ignore it, it's probably a poll and we can wait for the next one
-                return;
             }
         } else {
             // got a telegram with data in it. See if the src/dest matches that from the last one we sent
@@ -594,17 +594,23 @@ void EMSESP::incoming_telegram(uint8_t * data, const uint8_t length) {
             if (txservice_.is_last_tx(src, dest)) {
                 LOG_DEBUG(F("Last Tx read successful"));
                 txservice_.increment_telegram_read_count();
-                txservice_.send_poll();
-            } else {
+                txservice_.send_poll(); // close the bus
+                tx_successful = true;
+            }
+        }
+
+        // if Tx wasn't successful, retry
+        if (!tx_successful) {
             // the telegram we got wasn't what we had requested
             // So re-send the last Tx and increment retry count
             uint8_t retries = txservice_.retry_tx(); // returns 0 if exceeded count
             if (retries) {
-                    LOG_ERROR(F("Last Tx read failed. Retrying #%d..."), retries);
+                LOG_ERROR(F("Last Tx operation failed. Retrying #%d..."), retries);
             } else {
-                    LOG_ERROR(F("Last Tx read failed after %d retries"), txservice_.MAXIMUM_TX_RETRIES);
-                }
+                LOG_ERROR(F("Last Tx operation failed after %d retries. Ignoring request."), txservice_.MAXIMUM_TX_RETRIES);
             }
+
+            return;
         }
     }
 
@@ -825,7 +831,6 @@ void EMSESP::loop() {
         shower_.loop();    // check for shower on/off
         sensors_.loop();   // this will also send out via MQTT
         console_.loop();   // telnet/serial console
-        delay(ESP_DELAY);  // some time to WiFi and everything else to catch up, calls yield, and also prevent overheating
 
         // force a query on the EMS devices to fetch latest data at a set interval (1 min)
         if ((uuid::get_uptime() - last_fetch_ > EMS_FETCH_FREQUENCY)) {
@@ -833,7 +838,7 @@ void EMSESP::loop() {
             fetch_device_values();
         }
 
-        // helps ease wifi outages
+        // helps ease wifi dropouts effecting MQTT and Telnet services
         // https://github.com/esp8266/Arduino/blob/e721089e601985e633641ab7323f81a84ea0cd1b/cores/esp8266/core_esp8266_wiring.cpp#L41-L57
         delay(1);
     }

src/emsesp.h

@@ -47,7 +47,6 @@
 #include "roomcontrol.h"
 
 #define LOG_TRACE_WATCH_NONE 0 // no watch set
-#define ESP_DELAY 1
 
 namespace emsesp {
 

src/main.cpp

@@ -24,5 +24,4 @@ void setup() {
 
 void loop() {
     emsesp::EMSESP::loop();
-    yield();
 }

src/mqtt.cpp

@@ -369,7 +369,7 @@ void Mqtt::show_topic_handlers(uuid::console::Shell & shell, const uint8_t devic
         return;
     }
 
-    shell.print(F(" These MQTT topics are registered: "));
+    shell.print(F(" Subscribed MQTT topics: "));
     for (const auto & mqtt_function : mqtt_functions_) {
         if (mqtt_function.device_id_ == device_id) {
             shell.printf(F("%s "), mqtt_function.topic_.c_str());

src/sensors.h

@@ -69,7 +69,7 @@ class Sensors {
 #ifdef WEMOS_D1_32
     static constexpr uint8_t SENSOR_GPIO = 18; // Wemos D1-32 for compatibility D5
 #else
-    static constexpr uint8_t SENSOR_GPIO = 14;
+    static constexpr uint8_t SENSOR_GPIO = 14; // D5 is LED on wemos lolin D32, so use GPIO14
 #endif
 #endif
 

src/system.cpp

@@ -144,7 +144,7 @@ void System::start() {
     // register MQTT system commands
     Mqtt::subscribe("cmd", std::bind(&System::mqtt_commands, this, _1));
 
-// RTC state variables - onky for ESP8266
+// RTC state variables - only for ESP8266
 #if defined(ESP8266)
     state_.registerVar(&reset_counter_); // we send a pointer to each of our variables
     state_.registerVar(&safe_mode_);

src/telegram.cpp

@@ -388,10 +388,9 @@ void TxService::send() {
         return;
     }
 
-    // if there's nothing in the queue to send
-    // optionally, send back a poll and quit
+    // if there's nothing in the queue to transmit, send back a poll and quit
     if (tx_telegrams_.empty()) {
-        // send_poll(); // TODO commented out poll for now. should add back when stable.
+        send_poll();
         return;
     }
 
@@ -409,7 +408,7 @@ void TxService::send_telegram(const QueuedTxTelegram & tx_telegram) {
     // build the header
     auto telegram = tx_telegram.telegram_;
 
-    // src - set MSB if its Junkers/HT3
+    // src - set MSB if it's Junkers/HT3
     uint8_t src = telegram->src;
     if (ems_mask() != EMS_MASK_UNSET) {
         src ^= ems_mask();
@@ -484,7 +483,10 @@ void TxService::send_telegram(const QueuedTxTelegram & tx_telegram) {
 
     // send the telegram to the UART Tx
     EMSUART_STATUS status = EMSuart::transmit(telegram_raw, length);
-    //LOG_TRACE(F("Tx: %s"), Helpers::data_to_hex(telegram_raw, length).c_str());
+#ifdef EMSESP_DEBUG
+    LOG_TRACE(F("Tx: %s"), Helpers::data_to_hex(telegram_raw, length).c_str());
+#endif
+
     if (status != EMS_TX_STATUS_OK) {
         LOG_ERROR(F("Failed to transmit Tx via UART. Error: %s"), status == EMS_TX_WTD_TIMEOUT ? F("Timeout") : F("BRK"));
     }
@@ -610,9 +612,12 @@ void TxService::remember_tx(const uint8_t * data, const uint8_t length) {
 uint8_t TxService::retry_tx() {
     if (++retry_count_ == MAXIMUM_TX_RETRIES) {
         reset_retry_count();             // give up
-    } else {
-        add(telegram_last_, telegram_last_length_); // add the last Tx telegram to the tx queue, at the top
+        increment_telegram_fail_count(); // another Tx fail
+        return 0;
     }
+
+    add(telegram_last_, telegram_last_length_); // add the last Tx telegram to the tx queue, at the top
+
     return retry_count_;
 }
 

src/telegram.h

@@ -270,6 +270,14 @@ class TxService : public EMSbus {
         telegram_read_count_++;
     }
 
+    uint16_t telegram_fail_count() const {
+        return telegram_fail_count_;
+    }
+
+    void increment_telegram_fail_count() {
+        telegram_fail_count_++;
+    }
+
     uint16_t telegram_write_count() const {
         return telegram_write_count_;
     }
@@ -312,6 +320,7 @@ class TxService : public EMSbus {
     const std::shared_ptr<const Telegram> telegram_last;             // copy of last telegram
     uint16_t                              telegram_read_count_  = 0; // # Tx successful reads
     uint16_t                              telegram_write_count_ = 0; // # Tx successful writes
+    uint16_t                              telegram_fail_count_  = 0; // # Tx unsuccessful transmits
 
     uint8_t retry_count_ = 0; // count for # Tx retries
 

src/thermostat.cpp

@@ -1535,7 +1535,6 @@ void Thermostat::console_commands(Shell & shell, unsigned int context) {
                                            } else {
                                                set_temperature(atof(arguments.front().c_str()), HeatingCircuit::Mode::AUTO, hc);
                                            }
-
                                        });
 
     EMSESPShell::commands->add_command(

src/version.h

@@ -1 +1 @@
-#define EMSESP_APP_VERSION "2.0.0a12"
+#define EMSESP_APP_VERSION "2.0.0a13"