Windows Azure AppFabric Service Bus Future – PDC10 Session Review

Clemens Vasters – Principal Technical Lead @ Windows Azure Team

 

Service Bus Today

  • Naming fabric or Structure that sits out on the internet that provides us to create a namespace
  • When we create a namespace for the service bus, Microsoft provisions a tenant in a multi-tenant value fabric for us, where we can create endpoints, message buffers, and communicate through those.
  • Around this routing fabric we have several nodes where we connect and that’s is where the fabric is being used.
  • Use Case Samples:
    • Message Queues: Wherever you need some application queue
      • Airline that is using this to check-in and synchronize devices using this
    • Outside Services: Whenever you need to provide a public facing service, but your internal service
    • MultiCast (Eventing)
    • Direct Connect (Bridge two environments through Firewalls and NAT.

A look into the Future

  • Service Bus Capability Scope
    • Connectivity (currently in the CTP)
      • Service Relay
      • Protocol Tunnel
      • Eventing
      • Push
    • Messaging (currently in the CTP)
      • Queuing
      • Pub/Sub
      • Reliable Transfer
    • Service Management
      • Naming
      • Discovery
      • Monitoring
    • Orchestration (Workflow)
      • Routing
      • Coordination
      • Transformation

 

  • Service Bus October 2010 Labs (labs release not parity with production services)
    • New and Improved
      • Load Balancing
        • I need to be able to take multiple listeners and host them on Service Bus and have load balancing on them
        • This is not possible on the production environment, and in the Labs we have a new protocol called AnyCast to provide this.
      • Richer Management
      • Durable Message Buffers
        • We want to be more queues and less about message buffers
    • In this October 2010 Labs Release we don’t have:
      • HttpListeners
      • Capabilities in the Service Registry
      • Missing some composition capabilities
      • Missing all the bindings
    • The purpose is to gather feedback about this new protocol

 

  • Durable Message Buffers
Version Storage TTL Capacity Messages
Production In-Memory 10 mins 2 MB 60KB
Labs Durable and Replicated Storage No Limit 100 MB 256KB
    • Same Lightweight REST protocol
    • Long pooling support
    • Future Releases
      • Reliable Transfer protocol options
      • Higher throughput transport options
      • Volatile buffers for higher throughput
    • Currently the same protocol backed up with a REAL Queuing system that is reliable and capacity.

  • Listener Load Balancing
    • Connection point management separate from listener
      • Explicit Management of service bus connection points to the service bus
      • We gain the ability to know if someone is listening or not.
    • Multiple listeners can share the same connection point
    • Load balancing and no single point of failure
    • Sticky sessions

 

  • Session Multiplexing
    • One socket per listener
    • Optimized for short sessions, and short messages
      • The latency for setting up that session is now very short
    • No extra round-trip to set up a session
    • Note: The purpose is to reduce latency

  • Explicit Streaming Support (Default model in production Today)
    • One socket per client
    • Optimized for long sessions, streaming and very large messages
    • Lightweight handshake, mostly ‘naked’ end-to-end socket afterwards
    • Note: This will be made available later into the Labs

 

  • Protocols
    • Currently allowed
      • NMF (net.tcp)
      • Http(S)
    • Currently thought
      • FTP
      • SMTP
      • SMS
      • Other
        • Candidate Example of those:  XMPP, … (not planned but thought as possible candidates)

  • Messaging
    • Reliable, transacted, end-to-end message, transfer of message sequences
    • Local Transactions
    • Batches are transferred completely or not at all

  • Service Bus Pub/Sub – Topics
    • Service Bus Topics
      • Think of as a message log with multiple subscriptions
    • Durable message sequences for multiple readers with per-subscription cursors
    • Pull Model consumption using messaging protocols
    • Service Bus manages subscription state

  • Service Bus Pub/Sub – Eventing
    • Service Bus Events
      • We already have it with neteventrelaybinding
    • Multicast event distribution
    • Push delivery via outbound push or connectivity listeners
    • Same subscription and filter model as topics
    • Considering UDP for (potentially lossy) ultra-low latency delivery

  • Service Bus Push Notifications
    • Distribution points allow for push delivery to a variety of targets
    • Built on top of pub/sub topic infrastructure
    • Protocol adapters for HTTP, NET/TCP, SMS, SMTP, …
    • Subscription-level AuthN and reference headers

  • Monitoring
    • Monitor of Service Bus state
    • System events exposed as topics with pull and push delivery
    • Events can flow into store for analytics
    • Ad-Hoc queries into existing resources and their state

  • Discovery
    • Discovery of Service Bus resources and resource instances
    • Eventing and query mechanisms similar to monitoring
    • Service Presence
    • Custom Taxonomies

What can you do know. Use AppFabric Service Bus Labs

Building Offline Applications using Sync Framework and SQL Azure – PDC10 Session Review

Nina (Ling) Hu – Program Manager @ Sync Framework Team

Key Takeaways:

  • Offline applications  have lots of benefits comparing to online applications especially for mobile devices and cloud services
  • Sync Framework, SQL Azure and Windows Azure provide a data platform that makes it very easy to build offline applications and sync services
  • Sync Framework caters for offline on any client platform, and provides better end-to-end support for Windows, Silverlight and Windows Phone 7 clients.

Why to Build Offline capable Apps?

  • Offline/Cached mode apps enjoy lots of value
    • Client apps get better UX through
      • Lower latency – data access doesn’t  require round-trips
      • Higher availability – App still runs if server is unreachable
    • Network utilization is reduced – most data access is local
    • Servers gain better ability to schedule work asynchronously
  • Benefits of cached mode are magnifies on the internet because
    • Server/service is further away
    • Network is less tuned/reliable
    • Lowers the need of scaling you app

Overview for Sync Framework v4

  • Capabilities where extended to other platforms like:
    • Windows
      • SQL Server Express
      • SQL Compact
    • Silverlight
      • Isolated Storage
      • Other Storage
    • Windows Phone 7
      • Isolated Storage
      • Other Storage
    • Windows Mobile
      • SQL Compact
    • Browser 7 / HTML5
      • HTML5 Stores
    • iPhone / Any Client
      • SQL lite
      • Other Stores
  • How was this achieved?
    • Using a single protocol for doing the communication and Synchronization. ODATA
  • What platforms have Client API support?
    • Windows
    • Silverlight
    • Windows Phone 7
  • All other platforms have minimal client & store requirements
    • Basically there are made available a set of samples to show how to implement those.
    • Over time there will be provided some tools to generate the code needed to integrate with you platform

Offline Applications Architecture

  • On the Server
    • ODATA Sync Framework
    • Sync Logic
    • SQL Azure Provider
    • Business Logic
      • This is the only part needed to be written for our application
    • SQL Azure

 

    • Steps to Build a Sync Service
      1. Provision the Database to use Sync Framework
        1. Launch the SyncSVCUtilHelper.exe
        2. Select the configuration file for the Synchronization
          1. Select the Name of the configuration file
          2. Provide the name of the database
          3. Create a Sync Scope
            • Define if we enable filtering or not
          4. Select the Tables/Columns/Rows that you want to define the scope
        3. Provision the Database
          1. Select the configuration file
          2. Next
        4. Code Generation
          1. Select the Configuration file
          2. Specify the Side that you want the file to be Generated (Server/Client)
          3. Specify the Output directory
          4. Select the Language (CS/VB)
      2. Create the Sync Service
        1. Open VS
        2. Create a new ASP.NET Project
        3. Add Reference to Microsoft.Syncronization.Service.dll
        4. Add the Code generated by the Tool?
    • Note: When you open the Service in the Browser if you add the $diag you’ll get a page with the diagnostics of the service, and see if everything needed is in place

 

    • Overview
      • Data Store
        • Expose data from SQL Azure or SQL Azure
      • Host
        • Hosted on Windows Azure or IIS/Windows Server
        • Exposed using WCF Sync endpoint
      • Protocol
        • Expose data for synchronization via a protocol and allows third-parties to build offline clients
      • Features
        • Support business logic
        • Custom authentication / authorization
        • Filtering
      • Tooling
        • Provide a Tooling wizard experience to configuring server and client

 

  • On the Client
    • Silverlight Offline Application
      • Only this is needed to be developed
    • Collections
    • Isolated Storage
    • Cache Controller
    • OData Sync Proxy
    • Isolated Storage Provider

 

    • Support for Silverlight 3 and 4

 

    • Steps to build a Sync Framework Client
      1. Create a new Silverlight Application
      2. Add Reference Microsoft.Synchronization.ClientService.dll
      3. Add Reference System.ComponentModel.DataAnotation.dll
      4. Add Reference System.Windows.Controls.Data.dll
      5. Open the Tool SVCSyncUtilHelper.exe
      6. Choose the previous Configuration file
      7. Select the Side you want to generate (Isolated Storage client)
      8. Select the Language (CS/VB)
      9. Add the Files generated to your application

 

  • On the Client for other Platforms
    • Track changes on the local storage (samples for common stores)
    • Implement the client-side sync proxy (samples for sync proxies)

 

  • Protocol
    • OData Protocol & Sync
      • Apply the principals of OData to the problem of data-sync
      • Standardize on protocol not components
      • Minimal client sync logic and algorithms
      • Service manages sync keeping client simple
      • Provide samples to how to consume protocol for any platform
      • Provide components for richer experience in Silverlight & Windows Phone 7
      • Full interop details defined in the SDK

Roadmap

  • Sync Framework 4.0 CTP is available now.
  • SQL Azure Data Sync CTP 1 its for the Cloud to Cloud Service
  • SQL Azure Data Sync CTP 2 will support On-premise to Cloud Sync
  • Sync Framework 4.0 RTW will be available on 2011

More resources

Understanding Windows Azure Connect – PDC10 Session Review

Anthony Chavez – Director @ Windows Azure Networking Group

Introducing Windows Azure Connect

  • Secure network connectivity between on-premises and cloud
    • Supports standard IP protocols (TCP, UDP)
  • Example use cases:
    • Enterprise app migrated to Windows Azure that requires access to on-premise SQL Server
    • Windows Azure app domain-joined to corporate Active Directory
    • Remote administration and troubleshooting of Windows Azure roles
  • Simple setup and management

Roadmap

  • CTP release by end of 2010
    • Support connect from Azure to non-Azure resources
      • Supports Windows Server 2008 R2, Windows Server 2008, Windows 7, Windows Vista SP1, and up
  • Future releases
    • Enable connectivity using existing on-premises VPN devices

Closer Look

  • Three steps to setup Windows Azure connect
    1. Enable Windows Azure (WA) roles for External connectivity via service Model
      • Select only the roles that should be enabled for external onnectivity
    2. Enable local computers for connectivity by installing WA Connect Agent
    3. Configure/Manage your network policy that defines which Azure roles and which Azure computers can communicate.
      • defined using the Connect Portal
  • After the Configuration/Management of the Network Polity, Azure Connect automatically setups secure IP-level network between connected role instances and local computers
    • Tunnel firewall/NAT/s through hosted relay service
    • Secured via end-to-end IPSec
    • DNS name resolution

Windows Azure Service Deployment

  • To use Connect for a Windows Azure Service, enable one or more of its Roles
    • For Web & Worker Roles, include the connect plug-in as part of the Service Model (using .csdef file)
    • For VM Roles, install the connect agent in VHD image using Connect VM Install package
    • Connect agent will automatically be deployed for each new role instance that starts up
  • Connect agent configuration is managed through the ServiceConfiguration (.cscfg file)
    • One one configuration setting is required
      • ActivationToken
        • Unique per-subscription token, accessed from Admin UI
    • Several Optional settings for managing AD domain-join and service availability

Deployment

On-Premise

    • Local computers are enabled for connectivity by installing & activating the Connect Agent. It can be retrieved from:
      • Web-based installation link
        • Retrieved from the Admin Portal
        • Contains per-subscription activation token embedded in the url
      • Standalone install package
        • Retrieved from the Admin Portal
        • Enabled installation using existing software distribution tools
    • Connect agent tray icon & client UI, enables us to:
      • View activation state & connectivity status
      • Refresh network policy
    • Connect agent automatically manages network connectivity, by:
      • Setting up a virtual network adapter
      • “Auto-connecting” to Connect relay service as needed
      • Configuring IPSec policy based on network policy
      • Enabling DNS name resolution
      • Automatically syncing latest network policies

Management of Network Policy

    • Connect network policy managed through Windows Azure admin portal
      • Managed on a per-subscription basis
    • Local Computers are organized into groups
      • Eg. “SQL Server Group”, “Laptops Group”, …
      • A computer can only belong to a single group at a time
      • Newly activated computers aren’t assigned to any group
    • Windows Azure roles can be connected to groups
      • Enabled network connectivity between all Role instances (VMs) and local computer in the Group
      • Windows Azure connect doesn’t connect to other Windows Azure Roles
    • Groups can be connected to other Groups
      • Enabled network connectivity between computers in each group
      • A group can be ‘interconnected’ – enables connectivity within the group
      • Useful for ad-hoc & roaming scenarios
        • Eg. your laptop having a secure connection back to a server that resides inside the corp net

Network Behavior

  • Connect resources (Windows Azure role instances and external machines) have secure IP-level network connectivity
    • Regardless of physical network topology (Firewall / NATs) as long as they support outbound HTTPs access to Connect Relay service
  • Each connected machine has a routable IPv6 address
    • Connect agent sets up the virtual network address
    • No changes to existing networks
  • Communication between resources is secured via end-to-end certificate-based IPSec
    • Scoped to Connect Virtual network
    • Automated management of IPSec certificates
  • DNS name resolution for connected resources based on machine names
    • Both directions are supported (Windows Azure to Local Computer or vice-versa)

Active Directory Domain Join

  • Connect plug-in support domain-join of Windows Azure roles to on-premise Active Directory
  • Eg. Scenarios:
    • Log into Windows Azure using Domain Accounts
    • Connect to on-premise SQL Server using Windows Integrated Authentication
    • Migrate LOB apps to cloud that assume domain-join environment
  • Process:
    1. Install Connect agent on DC/DNS servers
      • Recommendation: create a dedicated site in the case of multiple DC environment
    2. Configure Connect plug-in to automatically join Windows Azure role instances to Active Directory
      • Specify credentials used for domain-join operation
      • Specify the target OU for Windows Azure roles
      • Specify the list of domain users / groups to add to the local administrators group
    3. Configure the network policy to enable connectivity between Windows Azure roles and DC/DNS Servers
    • Note: New Windows Azure role instances will automatically be domain-joined

Finally the recap of Windows Azure Connect

  • Enables secure network connectivity between Windows Azure and on-premise resources
  • Simple to Setup & Manage
    • Enabled Windows Azure Roles using connect plug-in
    • Install Connect agent on local computers
    • Configure network policy
  • Useful Scenarios:
    • Remote administration & troubleshooting
    • Windows Azure Apps Access to on-premise Servers
    • Domain-join Windows Azure roles

PDC2010 – Building Engaging Apps with Windows Azure Marketplace DataMarket Session

Adam Wilson – Program Manager @ Windows Azure DataMarket Team

Showing off different and Interesting integration scenarios with DataMarket

  • Facebook application that integrates with a Data.Gov DataSets
  • Usage of Windows Azure DataMarket Portal to explore the DataSets and also to show off the Tableau application to consume the DataSet from DataMarket

Integration Scenarios 

  • Choices:
    • Specific dataset versus generic
      • Influences the choice over the type of client technology to be used
        • WebRequest – need to parse the XML by hand
        • WCF Data Services client – ODATA client
        • DataMarket service proxy classes
    • Which DataMarket Account?

Building a simple Windows Phone 7 application

  • App that uses two DataMarket DataSets, one about weather forecast and another about the population density
    1. Choose the DataMarket DataSet
    2. Explore the DataSet
    3. Test the DataMarket DataSet
    4. Choose the Type of Integration to use
      • In this case the WebRequest is the one that is chosen
    5. Copy the Query URL
    6. Pass the Account Id and Account Key to the WebRequest
    7. Perform the Request
    8. Analyze the Response and Extract the XML elements that you need from the response.
  • Each content provider deliver a set of help resources like samples, and documentation about the respective DataSet that are accessible on the DataSets Page
  • Notes:
    • Can be built using WebRequest directly
    • Consider service-side
      • Filtering – Filter only what you need and cache your results on the server to lower the bandwidth
      • Projections – Do Projections on the Server-side in order to send only the elements that are interesting and needed by the client, reducing the used bandwidth

Next Steps: